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FINDING OUT FAST 


Policy decisions must frequently be made under pressure of time with lim- 
ited resources. This book is about skills and approaches to use in research 
designed to influence or inform policy against such a background. 


It is mot a handbook of tools designed to provide answers to all policy 
questions as quickly as possible. Rather, the book both explains and criti- 
cally reviews diverse methods for ‘finding out fast’. It aims to help readers 
develop sufficient skills to become competent investigators; to understand 
how to use research more effectively and how to commission and critically 
evaluate research done by others. 


Development takes place in a complex environment involving multiple 
agencies often with differing interests. The approaches in this book high- 
light the importance of advancing policy as an ongoing process rather than 
as a blueprint. This has implications for the way in which research is un- 
dertaken and conceptualized. The book will appeal to anybody involved in 
policy making and research related to policy aimed at developmental goals. 


Alan Thomas, academic editor 


This is a book for development practitioners, both beginners and those 
with more experience. Its main focus is reflection on and thinking through 
the nuts and bolts of development practice — thinking with paper, with 
people and organizations, with data. But it also has a great deal by way of 
practical insights and suggestions for those in the field. The book is unique 
and fills a glaring gap in the field of development practice. It deals with the 
dilemmas that every practitioner faces — the need to make decisions fast, 
provide advice on the basis of incomplete information, and navigate com- 
plex political waters. Usually one learns how to cope simply by coping! 
But Finding Out Fast shows that one can do better. Having to make deci- 
sions quickly or on the basis of less than perfect data doesn’t mean that one 
has to make sloppy decisions. 


The book is a fascinating tour through practical development dilemmas. 
Written by a group of authors with extensive practical experience in the 
field, it contains simple explanations as well as subtle nuances. Best of all, 
it is well-written and eminently readable. 


Gita Sen, Professor of Economics and Social Sciences, Indian Institute of 
Management 
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University, Milton Keynes, UK and the third is Marc Wuyts of the Institute 
of Social Studies in The Hague, Netherlands. 


ABOUT THE CONTRIBUTORS 


Stephanie Barrientos is a Senior Lecturer in Economic Development at the Uni- 
versity of Hertfordshire. She completed her degree and PhD at the University of 
Kent. Her main research interest is in Latin America, and on gender and agribusiness. 
She has published on women and the Chilean fruit export sector, and on the gender 
impact of global horticultural exports. 


Christine Blackmore is Lecturer in Environment and Development in the Open 
University’s Systems Department. Before joining the Open University in 1986 
Christine worked in Africa for 5 years (VSO, UNDP, Government of Sierra Leone) 
and taught in UK secondary schools for 4 years. She is a member of the Education 
for Sustainability Forum and the Education representative on the Executive Com- 
mittee of United Nations Environment and Development UK. 


Joanna Chataway is a Lecturer in Development Management at the Open Univer- 
sity, a position she took up in 1995. A core member of the team working on the 
Global Programme in Development Management, she previously worked for the 
Cambridge based consultancy firm Segal, Quince, Wickstead. As an academic and 
consultant she works on issues of institutional development, small business devel- 
opment and science and technology policy. She has worked in Russia, Central and 
Eastern Europe, South Africa and Central America. 


Joseph Hanlon is a Visiting Senior Research Fellow in the Open University Cen- 
tre for Technology Strategy. He is author or editor of 11 books on southern Africa, 
most recently Peace without profit: How the IMF blocks rebuilding in Mozam- 
bique ( Oxford, James Curry, 1996). 

Ray Ison joined the Open University as Professor of Systems in January 1994 
from the Faculty of Agriculture at the University of Sydney. His research 
specializations include: Systems research and development (R&D); development 
and evaluation of systemic, participatory R&D methodologies; process consultancy 
design and R&D evaluation for rural development; introducing problem based and 
experiential learning into curricula. 


Avril Joffe is a director of Labour Market Alternatives pty (Ltd.), a small Johan- 
nesburg-based company which works at the interface between industry develop- 
ment, industrial relations, human resource development and macro-economic policy. 
She was previously a lecturer in the Department of Sociology, University of the 
Witwatersrand and a co-director of the South African Industrial Strategy Project. 


Hazel Johnson is Lecturer in Development Studies at the Open University, has 
written widely for the undergraduate and postgraduate courses in development, 
and is a core member of the team working on the Global Programme in Develop- 
ment Management, She has a background in Sociology and Development Studies, 
and has carried out research in food production and markets, and food policy. Her 
current interests are in different aspects of institutional development. 

Geoff Jones is a lecturer in the School of Management at the Open University. A 
qualified accountant, his interests include the applicability of management con- 
cepts in differing cultural settings, knowledge management and the social and be- 
havioural aspects of accounting. 

Sue Mayer gained a PhD in cell biology from Bristol University in 1987. Since 
then she has lectured at Bristol University, worked for the RSPCA and spent 5 
years as Director of Science for Greenpeace UK. Since October 1995 Sue Mayer 


has been a freelance researcher and consultant in science and policy issues with 
special expertise in genetic engineering. 

Linda Mayoux is an independent consultant. She has done extensive research on 
gender and micro-enterprise development, co-operatives and micro-finance. One 
of her current interests is the development of participatory methods for gender analy- 
sis of micro-finance programmes. 


Chandan Mukherjee is Professor of Quantitative Methods and Director, at the 
Centre for Development Studies, Trivandrum, India, a research institute. He has 
been teaching statistics for over 20 years, at post graduate level and above, to stu- 
dents with varied disciplinary backgrounds. 

Bridget O’Laughlin teaches in the Population and Development Programme at 
the Institute of Social Studies in the Hague. From 1979-1992, she worked at Eduardo 
Mondlane University in Mozambique. She is currently researching on the conse- 
quences of the restructuring of migrant labour for rural livelihoods in Southern 
Africa. 


Stephen Potter joined the Open University in 1977 working on a number of trans- 
port research projects and also on an urban studies and a statistical sources course. 
In 1984 he left the Open University to be a freelance researcher and also worked 
with Dial-a-Ride in London. He rejoined the Open University in 1987 as a Re- 
search Fellow in the Technology Faculty. He has undertaken research work in a 
number of areas including transport, the management of design and urban studies. 


Chris Roche is currently team leader of the Gender and Learning Team in Oxfam’s 
policy department. Before this he worked for 10 years with the international NGO 
consortium ACORD in various guises and conducted research in West Africa on 
NGOs and in the UK on an Atlas of Social Protest. 


Ramya Subrahmanian is a PhD student in Development Studies at the Open Uni- 
versity. Her research focuses on universal primary education in rural India. She has 
worked as a consultant and trainer on gender-aware planning in development, work- 
ing with NGOs, Government of India and international organizations. She is also 
active in regional and national networks on women’s human rights. 


Alan Thomas has been Senior Lecturer in Systems at the Open University for 
some years and is a co-founder of the University’s Development Policy and Prac- 
tice group and a core member of the team working on the Global Programme in 
Development Management. He takes an action-oriented, case study approach to 
his research, where his main interests are co-operative development mainly in the 
UK and Europe, non-profit management and the role of NGOs in development and 
in environmental policy, mainly in Southern Africa. 


Philip Woodhouse is Lecturer in Environmental Management at the Institute for 
Development Policy and Management (IDPM), University of Manchester, and a 
director of postgraduate programmes in Environment and Development, and in 
Economics and Management of Rural Development. He has been engaged in re- 
search on rural resource management in Africa for 20 years, particularly in Mo- 
zambique, South Africa, and francophone West Africa. 


Mare Wuyts is Professor in Statistics and Development Economics at the Institute 
of Social Studies, The Hague, and teaches data analysis, statistics and econometrics. 
With Professor Chandan Mukherjee and Dr Howard White, he recently completed 
a book on econometrics and data analysis for developing countries. His research 
interests concern the analysis of structural adjustment, finance and development in 
Africa. 


This book is one component of the Open University course TU870 Capacities for 
Managing Development, a core course in the postgraduate Global Programme in 
Development Management. It was produced in conjunction with the TU870 course 
team and the contribution of the following course team members in the develop- 
ment and production of this book is gratefully acknowledged: 

Philippa Broadbent (Print Buying Controller) 

Suzanne Brown (Course Manager) 

Deborah Bywater (Project Controller) 

Joanna Chataway (Academic) 

Daphne Cross (Print Buying Co-ordinator) 

Debbie Dickinson (Secretary) 

Tony Duggan (Project Controller) 

Tom Hewitt (Academic) 

Hazel Johnson (Course Chair and Academic) 

Roy Lawrance (Graphic Artist) 

June McGowan (Secretary) 

Jane Moore (Editor) 

Janice Robertson (Editor) 

Dorcas Robinson (Researcher) 


Gita Sen (Professor, Indian Institute of Management, Bangalore; 
External Assessor) 


Jane Sheppard (Graphic Designer) 

Alan Thomas (Academic) 

David Wield (Academic) 

Gordon Wilson (Academic) 

The production of the course was made possible through generous support 
from the following: Catholic Fund for Overseas Development, Christian Aid, 


Cooperation for Development, Oxfam, Save the Children, Voluntary Service Over- 
seas. 


Details of this and other Open University courses can be obtained from the Course 
Reservations and Sales Office, PO Box 724, The Open University, Milton Keynes 
MK7 6ZS, United Kingdom: tel. (00 44) 1908 653231. 


For the availability of course components, contact Open University Worldwide 
Ltd., The Berrill Building, Walton Hall, Milton Keynes MK1 6AA, United King- 
dom: tel. (00 44) 858585, fax (00 44) 1908 858787, e-mail ouweng @open.ac.uk 


Alternatively, much useful course information can be obtained from the Open 
University’s website http://www.open.ac.uk 


IN\Gour 


AST wwasiemm INVESTIGATIVE 


SKILLS FOR 
POLICY AND 
DEVELOPMENT 


é Poblicatiles ALAN THOMAS 
TS loses twos ca-S Ot JOANNA CHATAWAY 
BN ASSOGATON Wir AND MARC WUYTS 


The Open University, Walton Hall, Milton Keynes MK7 6AA 

© The Open University 1998 

First published in 1998 

The opinions expressed are not necessarily those of the Open University. 


All rights reserved; no part of this publication may be reproduced, stored in a retrieval 
system, or transmitted in any form or by any means, electronic, mechanical, 
photocopying, recording or otherwise without either the prior written permission of the 
Publishers or a licence permitting restricted copying issued by the Copyright Licensing 
Agency Ltd., 90 Tottenham Court Road, London WIP OLP. This book may not be lent, 
resold, hired out or otherwise disposed of by way of trade in any form of binding or 
cover other than that in which it is published, without prior consent of the Publishers. 


) 


SAGE Publications Ltd 
6 Bonhill Street 
London EC2A 4PU 


SAGE Publications Inc. 
2455 Teller Road 
Thousand Oaks 
California 91320 


Sage Publications India Pvt Ltd 
32, M-block Market 

Greater Kailash - I 

New Dehli 110 048 


British Library Cataloguing in Publication Data 
A catalogue record for this book is available from the British Library 


Library of Congress catalog record available 


ISBN 0 7619 5836 3 (cased) 

ISBN 0 7619 5837 1 (pbk) 

Edited, designed and typeset by The Open University. 

Printed and bound by Scotprint Limited, Musselburgh, Scotland, UK. 


CONTENTS 
1 


INTRODUCTION 
Alan Thomas 


PARTI 
CONCEPTUALIZING POLICY-RELATED INVESTIGATION 


1 INFORMATION NEEDS AND POLICY CHANGE 19 
Stephen Potter and Ramya Subrahmanian 

2 BOUNDARIES FOR THINKING AND ACTION 41 
Chris Blackmore and Ray Ison 

3 GRABBING ATTENTION 67 
Joseph Hanlon 

PART Il 
THINKING WITH PAPER 

4 HOWTO DO A LITERATURE STUDY 87 
Stephanie Barrientos 

5 INTERPRETING INSTITUTIONAL DISCOURSES 107 
Bridget O’Laughlin 

PART 
THINKING WITH PEOPLE AND ORGANIZATIONS 

6 PEOPLE AS INFORMANTS 127 
Philip Woodhouse 

7 INVESTIGATION AS EMPOWERMENT: USING PARTICIPATORY METHODS 147 
Hazel Johnson and Linda Mayoux 

8 ORGANIZATIONAL ASSESSMENT AND INSTITUTIONAL FOOTPRINTS 173 
Chris Roche 

9 DILEMMAS OF RESEARCHING POVERTY 201 
Dina Abbott 

10 COMMUNICATING RESULTS 221 


Joanna Chataway and Avril Joffe 


PART IV 
THINKING WITH DATA 


11 THINKING WITH QUANTITATIVE DATA 237 
Chandan Mukherjee and Marc Wuyts 

12 ANALYSING INSTITUTIONAL ACCOUNTS 261 
Geoff Jones 

13 CRITICAL ISSUES IN USING DATA 289 
Sue Mayer 

14 CHALLENGING CASES 307 
Alan Thomas 
CONCLUSION: PERSONAL EFFECTIVENESS AND INTEGRITY 333 
Alan Thomas and Joanna Chataway 
REFERENCES 345 
ACKNOWLEDGEMENTS 361 


INDEX 362 


INTRODUCTION 


ALAN THOMAS 


This book is about research to inform policy and public action, particularly 
on development. It is aimed at development managers and others who are 
involved in policy investigation. Those doing academic research on policy- 
related issues should also find it useful, although it is not specifically ad- 
dressed to them. 


When I first suggested the title Finding out Fast there were two reactions. 
Most people were positive. They knew that policy decisions are often re- 
quired quickly, with incomplete data or with limited resources with which 
to obtain information. Hence, those involved in or learning about develop- 
ment policy need ideas and guidance on how to find out, evaluate and use 
relevant information fast. 


The other reaction was that the title was wrong, too slick or risked lending 
credibility to the idea that substantive research is generally an unnecessary 
waste of resources. The very idea of ‘finding out’ could sound as though 
the problem under investigation is more or less well understood and all 
that is needed is to ‘fill in the boxes’ with data. It could also imply that the 
investigator is extracting information from a situation without any involve- 
ment from those actually living or working in it. 


As for finding out fast, the book could easily appear to be promoting speed 
in relation to policy research as a good thing in itself. If rapid research is 
good, then more considered investigations may appear a luxury. It is 
already all too common to engage consultants to spend a very limited time 
investigating ‘the field’ at such a late stage in the policy development pro- 
cess that hardly any changes can be made to the proposed policy whatever 
the consultant’s findings (see Example A cited below from Staudt, 1991). 
One critic suggested that the only useful book on the subject would be a 
polemic against the ‘rapid rural appraisal’ (RRA) movement, pointing out 
that even the leading proponent of this approach, Robert Chambers, has 
recently expressed doubts about some of the ways it is used: ‘the word 
“rapid” ... has been used to legitimate brash and biased rural development 
tourism’ (Chambers, 1994a, p.1441). 


There are real dangers in certain methods being open to misuse and in the 
need for speed becoming an excuse for not allocating adequate resources 
to analysis and investigation. Nevertheless, speed is often required and 
delay is itself a policy decision. Whether you are implementing policy or 
trying to influence the actions of others, the other agencies involved will 
certainly not wait for your agency’s convenience. Thus the need to act as 
the situation or a politically imposed timetable demands may limit the ap- 
plicability of standard research methods which have self-imposed require- 
ments for sample size, statistical confidence or corroboration of evidence. 
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It is a reality for many development managers that investigations have to 
be done quickly — or if not, then to a timetable dictated by events outside 
their control. 


This book aims to help readers deal with such situations, where they may 
need to develop policy or make decisions quickly, about public action or 
their agency’s input into the policy process, with incomplete data or with 
limited resources with which to undertake an investigation. However, it is 
not a handbook extolling the virtues of a specific bag of tools which sup- 
posedly provides answers to all policy questions in as short a space of time 
as may happen to be available. Rather, the book both explains and criti- 
cally reviews the use of a variety of methods for ‘finding out fast’, in the 
belief that such methods can still be rigorous and that fast need not mean 


sloppy. 
The aims of the book can be summarized as follows: 


1 to help development managers to become competent investigators (not 
necessarily fully expert researchers), in the context of development as 
public action involving multiple agencies and of policy as process; 


2 for readers to understand how to use research better (including com- 
missioning and evaluating research); 


3 for readers to develop sufficient skills to enable them to do some of 
their own investigation, at least to a point that allows them to appreciate 
the research done by others; 


4 to introduce a range of generic analytical and investigative skills at a 
conceptual level, with emphasis on how the appropriateness of differ- 
ent methods derives from the needs of different contexts. Some practi- 
cal examples are provided; 


5 for each such skill area introduced, to present a checklist of points to 
bear in mind when using it or assessing an investigation which uses it. 


The rest of this Introduction is in six sections. The first explores the con- 
text of policy investigations with the aid of some illustrative examples. 
Sections 2 and 3 look more closely at the dangers of and the need for 
finding out fast, respectively. Section 4 explains the general approach taken 
to investigation in the book and Section 5 gives some additional points on 
investigation when it is specifically aimed at informing development policy 
and public action. The final section sets out how the rest of the book is 
organized. 


1 THE CONTEXT OF POLICY INVESTIGATIONS 


Here are a number of examples where policy is developing and choices 
have to be made (some of which appear as more detailed cases in later 
chapters). In each case there was an investigation, aimed at informing policy 
directly, at improving the quality of policy decisions, or at backing up a 
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case being made for a change in policy. Although the examples are deliber- 
ately chosen to illustrate the enormous variety of situations where there is 
a need for research to inform development policy, it is instructive to look 
for points of commonality between the various examples. 


EXAMPLE A GATHERING INFORMATION FOR PROJECT DESIGN 


A large international development agency has already undertaken a 
preliminary design for an integrated rural development project. ‘It 
aimed to provide heavily subsidized credit to small-scale farmers so 
that they would adopt a tobacco fertilizer package. Credit was to be 
delivered through a newly instituted co-operative society. Rural roads 
construction would enable farmers to market crops more easily. Nu- 
trition education and kitchen gardens would be developed for women.” 
The agency then puts together a project design team consisting of an 
anthropologist, an economist, a food scientist, two road engineers 
and an expert on finance. They are to provide information, on the 
local communities, on the costs of road construction, on appropriate 
varieties of vegetable, etc. This information is required for the de- 
tailed design of the project so that its implementation can be agreed 
by the national government and the required disbursement of aid funds 
can be confirmed by the agency’s headquarters. 

(taken from Staudt, 1991, pp.46-47) 


EXAMPLE B FROM ROUTINE INFORMATION TO 
INVESTIGATING CAUSES 


In Britain for many years, transport policy has consisted mainly of 
attempts by the Department of Transport on the one hand and various 
local authorities on the other to build roads to cope with the continu- 
ously rising demand for travel by car. Information has been routinely 
collected on trip lengths, types of destinations and frequency of trips, 
etc. so that trends in demand can be estimated. However, it has re- 
cently been suggested that policy should not just cope with demand 
but should attempt to affect travellers’ behaviour; perhaps policy 
should be designed deliberately to shape or even reduce demand in 
order to reduce congestion and pollution while maintaining accessi- 
bility and improving quality of life. This would imply the need not 
just for routine information gathering but for investigation of why 
people make the journeys they do and what controllable factors are 
likely to have an effect on demand. 


(See Chapter | for more on this example.) 
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EXAMPLE C INVESTIGATION TO INFLUENCE POLICY 


The Southern Okavango Integrated Water Development Project 
(SOIWDP) was a proposed engineering project developed by the 
Botswana government in the late 1980s. The Okavango Delta, in the 
north-west of the country, is one of the world’s greatest wetlands, 
world famous for its wildlife and hence important for tourism. It also 
provides livelihoods in an otherwise arid region for over 100 000 peo- 
ple of a variety of ethnic groups. The SOTWDP was to provide water 
for commercial irrigation, village-based flood recession agriculture, 
communities and livestock throughout the area, and particularly the 
town of Maun and the De Beers diamond mine at Orapa. 


When earth-moving equipment began to arrive in Maun in late 1990, 
anew local action group was formed which mounted a very effective 
local campaign against the SOIWDP. In parallel, local and South Af- 
rican conservationists contacted Greenpeace International, which in 
turn wrote to the Botswana government and on its invitation sent a 
team of three on a two-week study tour. Thus the Greenpeace team 
had a very brief period of investigation on the basis of which to try to 
influence the Botswana government. 


(See Chapter 14 for more on this example from a different point of view.) 


Some of the points in common between the three examples are as follows: 


1 Rather than being concerned solely with improving their own positions, 
the agencies involved in each case are intervening in social processes 
in order to try to work towards broad social (or environmental) goals 
external to the agencies themselves. 


In Example A, the agency’s project is presumably intended to improve 
the livelihoods of the small farmers, while in Example B the possible 
change to transport policy would clearly be designed to change peo- 
ple’s behaviour in a way which is not occurring voluntarily but which 
would be intended to improve their quality of life. Example C sees an 
international non-governmental organization (NGO) based in the North 
intervening in the affairs of a Southern country in the name of the envi- 
ronment. 


2 In each case multiple agencies are involved, including several other 
organizations and groups as well as the agency undertaking the investi- 
gation. 

The relationship between them can be of different kinds. In Example A 
the project team is seeking support from its agency’s headquarters and 
from the government of the country concerned, as well as having 
to deal with local community groups and interests. In Example B the 
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Department of Transport and the various local authorities are said to be 
working together, although there may be some differences of interest 
between particular local needs and considerations of overall effects at 
national level. In Example C, depending on the outcome of its team’s 
investigation, Greenpeace may well find itself opposed to the national 
government, although in alliance with certain local interests. 


The social goals being aimed at are subject to value-based conflicts. 


It is clear in Example C that Greenpeace’s environmental goals may 
conflict with certain views of the goals of Botswana’s national develop- 
ment. In Example B, saving energy and reducing pollution nationally 
may conflict with improving accessibility in a particular location. As 
for Example A, what the agency headquarters or its national staff sees 
as a good rural development project may differ from the priorities of 
the particular government concerned, while what different groups in 
the community want may be different again. It is not just that different 
interests clash, but views as to what constitute desirable social and de- 
velopment goals differ according to what weight is given to different 
values, as for example when improved material welfare may be ob- 
tained at the expense of freedom of choice. 


The decisions which may be informed by particular investigations are 
always part of the continuing development of policy so that it is best to 
think of policy as process. 


This means that not only is policy designed to change a given situation 
but the situation is changing anyway and giving rise to changing press- 
ures for changes in policy. The fact that policy is constantly developing 
in this way makes it useful to think of policy as itself a process. 


The alternative view, of policy as prescription, is not appropriate in 
development contexts. It is never the case that an agency simply comes 
to a ‘green field’ situation, finds out what is needed, and prescribes and 
implements policies designed on that basis. In Example A the agency is 
undertaking an investigation as part of its own process of designing a 
project. More generally, agencies tend to have policies already operat- 
ing, as with the policy of building roads to meet demand in Example B. 
And if a particular agency is indeed new to a situation — as in Example 
C - then there are generally other agencies already involved which are 
carrying out policies. This means you cannot assume things will remain 
Static for as long as it takes to do whatever investigation is deemed 
necessary. The situation may change as the policies of the other agen- 
cies involved impact upon it. 

Similarly, once an investigation is complete, an agency’s policies may 
be informed by it for some time, as may the actions of others, but both 
the policies and the actions of all concerned will continue to change and 
develop as events unfold and one reacts to another. 
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The above commonalities between the examples may be summarized by 
pointing out that they all involve development management (Thomas, 
1996a), where development is seen in terms of public action (Dréze and 
Sen, 1989; Wuyts, Mackintosh and Hewitt, 1992). Boxes 1 and 2 give defi- 
nitions of these terms, for reference. 


BOX 1 DEVELOPMENT MANAGEMENT 


... development management [is] the management of deliberate ef- 
forts at progress on the part of one of a number of agencies, the man- 
agement of intervention in the process of social change in the context 
of conflicts of goals, values and interests. (... in this view, develop- 
ment management is a process or an activity that can take place any- 
where, not just in developing countries.) 

(Thomas, 1996, p. 106) 


BOX 2 PUBLIC ACTION 


Public action is not ... just a question of public delivery and state 
initiative. It is also ... a matter of participation by the public in the 
process of social change. 
(Dréze and Sen, 1989, p, 259) 


By public action we mean purposive collective action, whether for 
collective private ends or for public ends (however defined). ... [A] 
wide definition of public action ... include[s] ... the actions, not only 
of states and organizations which claim to seek to promote the public 
good or assist the disadvantaged, but also clubs of the powerful who 
seek to influence the public sphere for their own ends. 


(Mackintosh, 1992, p. 5) 


There are some final points to note about the above examples. First, who is 
doing the research is a political question which affects how the research 
can be done and how the results are likely to be utilized. The investigator 
cannot think of herself or himself as standing apart from what is being 
investigated or think of the investigation as an exercise in objectivity to be 
kept separate from policy actions. It does matter whether the investigator 
is male or female, or is perceived as an agent of a powerful agency, and so 
on, In particular, the investigator's relationship with the agencies involved 
is an important part of the context. This relationship can be of various 
kinds. In Example A the investigators will be trying to influence their own 
agency (and an outside agency — the government) without having direct 
power to implement. In other cases an investigator could be intending to 
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use research results to inform and implement policies directly; or carrying 
out a consultancy for one of a number of agencies; or representing a spe- 
cific interest group trying to influence another, perhaps more powerful, 
agency. 


In each case the investigations to be carried out are not only action- 
or policy-oriented but they are themselves part of the process of policy 
development and public action, not separate activities. In Example A, 
the way the investigation is carried out will lead to certain expectations 
on the part of those being investigated. In Example C, the way the study 
tour came about and the fact that it was being carried out by Greenpeace 
had direct implications. The results were communicated jointly by 
Greenpeace and the Botswana government and this may have had more 
impact than the content of the communiqué. In general investigations are 
part of the policy process. However, such investigation is equally demand- 
ing of rigour as other academic research. Box 3 summarizes the frame- 
work suggested for placing in context the type of investigations covered in 
this book. 


BOX 3 THE CONTEXT OF POLICY INVESTIGATIONS 


Investigations aimed at informing or influencing policy or public 

action can be usefully viewed in the following framework: 

1 The agencies involved are intervening in social processes. 

2 They are aiming at goals external to the agencies themselves. 

3 This means development as public action involving multiple agen- 
cies. 

4 Development goals tend to be subject to value-based conflicts. 

5 Such situations are best understood in terms of policy as process. 


6 The investigator's relationship with the agencies has to be consid- 
ered. 


7 Finally, one should see investigations as part of the policy process. 


2 THE DANGERS OF FINDING OUT FAST 


As pointed out at the very beginning of this Introduction, the idea of find- 
ing out fast should not be adopted uncritically. I will divide the problems 
into dangers of a ‘finding out’ approach and dangers of (over) rapid re- 
search. In each case the same examples (A to C, outlined above) are used 
to provide illustrations. In fact you may already have noticed some of the 
dangers which I am about to bring out when the examples were being dis- 
cussed above. 
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2.1 DANGERS OF A ‘FINDING OUT’ APPROACH 


Let us look first at some dangers of a ‘finding out’ approach (Box 4 sum- 
marizes these). 


1 ‘Finding out’ sounds very much as though one knows in advance what 
is to be found out. In other words, the problem under investigation is 
conceptualized in advance. But conceptualization should not remain 
fixed or contrary evidence may be ignored. 


In Example A, it appears that the outline of the project is already fixed 
so that the investigation is simply to find out things like which vegeta- 
ble varieties are appropriate. ‘Provid[ing] information on the local com- 
munities’ sounds more open-ended, but this formulation fails to make 
explicit what underlying assumptions may have been made about the 
nature of social relations in these communities. As far as gender rela- 
tions are concerned, some assumptions are clearly implied by the fact 
that the project appears to be designed with a built-in women’s element 
including nutrition education and vegetable gardens. While this assump- 
tion may be well founded, thinking of the investigation in terms of sim- 
ply ‘finding out’ or ‘providing information’ may mitigate against look- 
ing for evidence that could either confirm or cast doubt on the underly- 
ing way in which aspects of the problem, such as the pattern of gender 
relations, have been conceptualized. 


2 ‘Finding out’ seems to imply that the form of the problem is already 
known, so that one knows which boxes have to be filled in with data. 
This may be appropriate for administration but ‘filling in the boxes’ is 
not sufficient to inform development interventions, which are aimed at 
assisting a process of change in a positive direction. 


Thus in Example B we see how continuing to fill in boxes with data 
about trip lengths etc. will build up data series over time which are 
consistent in form from year to year, and thus help to administer a road- 
building programme. However, once it is decided to use policy to man- 
age demand, that is to try to change the direction of development, then 
simply finding out is not sufficient. There is a need to build up a picture 
of motivations and interlocking causes, and in such a case there is likely 
to be conflicting evidence needing to be weighed up so that the notion 
of ‘finding out’ may lead to over-simplification. 


3 The notion of gathering or extracting information can imply no involve- 
ment from those in the situation. 
Again, the idea of ‘provid[ing] information on the local communities’ 
in Example A seems to imply observing or measuring them from out- 
side. Quite apart from the dubious ethics of such an approach, the fact 
that people in any situation have their own aims and aspirations and 
will react to a policy initiative accordingly makes it important for such 
an investigation to be interactive if not fully participative. In fact it will 
be argued in several chapters of this book that investigations always 
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involve the people in the situation under investigation in some way, 
whether the methods are labelled participative or not, and the question 
is on what terms they are involved. 


BOX 4 DANGERS OF A ‘FINDING OUT’ APPROACH 


‘Finding out’ implies the conceptualization of the problem under 
investigation is fixed. 


‘Filling in the boxes’ with data is not sufficient to inform develop- 
ment interventions. 


‘Gathering’ or ‘extracting’ information can imply no involvement 
from those in the situation. 


2.2 DANGERS OF (OVER) RAPID RESEARCH 


There are also dangers in assuming that doing research quickly is a good 

thing in itself (see Box 5): 

1 The fact that teams of professional researchers can be assembled and 
paid to produce reports in very short spaces of time means ‘proper’ 
research can appear a luxury and resources may not be made available. 
In the case in Example A, for example, Staudt (1991, p. 46) describes 
how only 10 working days were specified for the project in the research- 
ers’ contracts, of which two were allocated to visit ‘the field’. 


The existence of known and labelled rapid research techniques may 
encourage the idea that research can be done extremely quickly. How- 
ever, as you will see in Chapters 6 and 7, many of the techniques used in 
rapid rural appraisal (RRA) and particularly in participative rural ap- 
praisal (PRA), although not as time-consuming as traditional anthropo- 
logical research, still require considerable time and resources if they 
are to be carried out rigorously. 


2. Research thought to be shallow can be rubbished. 


The brief study tour in Example C ended with a press communiqué 
agreed jointly by the Greenpeace team and the Botswana government. 
Nevertheless, Greenpeace has been repeatedly criticized in Botswana 
since then for opposing a well-worked-out scheme on flimsy evidence. 
NGOs and other interest groups lobbying to change official policies of 
governments or intergovernmental agencies such as the World Bank 
can all too easily run into this type of criticism, especially as they often 
have to react quickly to policy changes announced by those official 
agencies. This makes it all the more important to show clearly the rigor- 
ous basis of investigation on which alternatives put forward are based. 

3 Certain specific rapid research techniques may be used unthinkingly, in 
an over-standard way including the advance definition of problems from 
outside. 


FINDING OUT FAST: INVESTIGATIVE SKILLS FOR POLICY AND DEVELOPMENT 


Here I am reminded of the possibly apocryphal tale in which a research 
team arrives in a remote village to be greeted by an old woman who 
makes it known through their interpreter that she expects them to inter- 
view her as she is a ‘key informant’. It is not only the techniques of 
RRA that can become over-standardized. In general the tendency to 
define problems from outside tends to combine with the first of the 
dangers of a ‘finding out’ approach explained above, viz. that of fixing 
one’s conceptualization of a situation in advance. This was another criti- 
cism advanced against Greenpeace in Example C. In this view, since 
the NGO was already convinced that all engineering development was 
bad for the environment, it needed only to apply standard techniques of 
environmental impact assessment to this particular case. Though this 
criticism may be ill-founded, in order to counter it one would have to 
demonstrate clearly that the techniques used were chosen as appropri- 
ate after an open-ended appraisal of the overall situation and that there 
was a real possibility of a contrary result from the investigation. 


Having to complete an investigation quickly may mean effort may be 
channelled into what is easier to investigate — towards topics which are 
less challenging or towards people who are easier to reach and to com- 
municate with. This can all too easily lead to a bias against those who 
are poor and powerless, which often includes women, those in remote 
or inaccessible areas, children and old people. It may mean not broach- 
ing issues of social differentiation and power relations, and failing to 
challenge assumptions — particularly if that might be politically diffi- 
cult as well as requiring more time to build up evidence. 

In Example A, it appeared that an assumption of the preliminary project 
design was that the project should concentrate on those farming be- 
tween 5 acres and 20 acres of land. It was therefore not expected that 
time should be spent finding out about the majority - women farmers 
with plots much smaller than 5 acres and with menfolk migrating for 
wage employment elsewhere — or to investigate the likely effects of the 
proposed project on them. It would have been particularly difficult and 
time-consuming to investigate the absent men. 


The combination of reduced resources (especially time) plus the expec- 
tation that rapid research includes some kind of direct field observation 
involving outsiders can mean that other research methods such as liter- 
ature study and the analysis of secondary data may be squeezed out. 


It is all too easy to assume that there is no prior knowledge of a particu- 
lar case. However, there are often relevant studies either by academics 
or by other agencies lying unread. Perhaps the fact that certain methods 
of field observation and interviewing are called ‘rapid research’, com- 
bine with the apparent uncertainty of getting results from a literature 
study or from the analysis of incomplete data, make the latter appear 
rather risky undertakings when there is a fixed and very limited timescale. 
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BOX 5 DANGERS OF (OVER) RAPID RESEARCH 


“Proper’ research can appear a luxury. 
2 Research which appears shallow can be rubbished. 
Certain specific rapid research techniques may be used unthink- 


ingly. 

Effort may be channelled into what is easier to investigate. 
Literature study and analysis of secondary data may be squeezed 
out. 


Thus, although the context may require investigation to be done to a 
timescale outside one’s control, one has to be aware of the dangers of as- 
suming that research done quickly is necessarily the best research. In par- 
ticular, the necessity for rigour is if anything intensified when finding out 
fast. 


3 THE NEED FOR FINDING OUT FAST 


Despite the dangers of assuming fast is best, any investigation which is 
aimed at influencing development policy or informing public action gen- 
erally has to be done with limited human and financial resources as well as 
to a timescale which is outside one’s control. 


In Examples A and C both these constraints were operating. In Example A, 
the researchers were contracted for 10 days only, presumably because the 
agency had limited resources available for this part of the project develop- 
ment process. Similarly in Example C Greenpeace decided that the situa- 
tion warranted a three-person team with its international travel and other 
costs, for a two-week period. Generally any agency faced with the need for 
evidence, whether to strengthen its own project design or to make a case to 
another agency, has to decide how much of its resources should be com- 
mitted. This implies weighing up the benefit to be obtained from the likely 
results of an investigation against the cost of the resources needed to ob- 
tain them, which could otherwise be put to use elsewhere. 


If financial resources are the main constraint, then there is a choice as to 
how fast a piece of research should be done. If resources are extremely 
limited, there is a lot to be said for a concentrated effort giving some re- 
sults quickly, perhaps at least an interesting reconceptualization. Taking a 
long time over an investigation may lead to expectations that the work will 
have been extremely detailed when resources were not available for that. 
On the other hand, it may be worth spreading small amounts of available 
time over a longer period if that is the way to get access to a number of 
very busy people, to obtain various documents as they are suggested by the 
unfolding investigation, and to fit in visits to different locations. 


FINDING OUT FAST: INVESTIGATIVE SKILLS FOR POLICY AND DEVELOPMENT 


Whether or not a policy investigation is actually done fast, it is likely to be 
done to meet someone else’s deadline. This may be short and suddenly 
imposed. For example, when challenging a decision, the opposition case 
has to be put forward at the time if implementation is to be prevented, so 
that evidence for that case may have to be prepared almost immediately. 
The Greenpeace team in Example C were up against a deadline of this 
type. In other cases, the deadline may be known well in advance. If the 
results of a piece of evaluative research are to be used to argue for the 
continuation of a certain project then the case has to be put at the right time 
to fit in with the budget cycle. Although there should be enough notice to 
set up monitoring or other investigations in good time, there will always 
be limited resources, those who will undertake the evaluation will have 
other responsibilities, and the research work may well end up having to be 
done fast. Though not an evaluation, the investigation in Example A seems 
to have been rather like this. 


One should also realize that coming up with some results fast does not 
have to be the end of the investigative process. The need for ideas and 
evidence to input to a particular policy discussion or decision is not the 
end of the matter. Just as the development and implementation of policy is 
an ongoing process so too investigation is a parallel process which in- 
forms both specific interventions and reviews of policy. Thus in Example 
B, I can envisage particular investigations needing to be done quickly for 
a particular meeting or decision, while debate over the future direction of 
transport policy continues, informed by the cumulative results from such 
investigations and in turn informing the way such investigations are car- 
ried out, over an indefinite period. 


There is of course an apparent conflict between finding out fast and being 
able to back up your findings with firm evidence. The need for rigour 
cannot be emphasized too much. Readers who are also development man- 
agers are likely to have pressures on them to act quickly and may have 
begun this book in the hope that finding out fast will enable them to act 
even more quickly and on a secure basis. However, finding out fast does 
not mean you can necessarily do everything you would like in as short a 
time as is available. One answer to this problem is to recognize that in 
policy investigation rigour implies selectivity about the object of study. 
The need for rigour implies using the time available to challenge the most 
basic assumptions or to look for evidence in the areas of greatest uncer- 
tainty. There is no point in a broad search for all kinds of data if lack of 
time means none of it can be cross-checked. 


To summarize this section, in general any policy-oriented investigation 
has to be undertaken if not necessarily with ultra-rapidity then certainly to 
meet deadlines set by others or by limitations of resources, rather than to 
suit the ideal requirements of particular research methods. Maintaining 
rigour in this context implies selectivity about the aspects to be focused on 
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within the set timescale; given that investigation is itself an ongoing pro- 
cess, other aspects may be investigated later if necessary. 


4 THE APPROACH TO INVESTIGATION IN THIS BOOK 


Although development policy and public action are the context, the book 
will be focused on skills and methods for investigation rather than on the 
policy process itself. This section explains the approach taken to investiga- 
tion by means of some general points about investigation, before the fol- 
lowing section goes on to some additional points specific to investigation 
for policy and public action. Many of the following points come directly 
from some of the arguments made above. 


1 Investigation is considered to be a more fruitful notion than that of in- 
formation gathering. Investigation evokes the image of a detective rather 
than a note-taker. It implies asking about how and why things happen 
as well as about what has happened. It may be necessary to spend time 
checking the answers to a set series of questions but investigation im- 
plies going beyond this, worrying about loose ends and following up 
leads. 

2 Rigour is always crucial in any investigation. This is equally the case 
whether one is undertaking academic research or working in the practi- 
cal context of development policy. Rigour means being able to show 
that one has enough evidence to justify one’s conclusions, that the evi- 
dence has been obtained properly and that contrary evidence has been 
sought but either not found or found to be relatively unconvincing. 


3 Investigation means conceptualizing and reconceptualizing, not just 

gathering information to fit pre-defined categories. A good analogy is 
with the way a journalist finds a ‘story’ through which to tell the results 
of an investigation. One may start off expecting to build up a particular 
story but end up finding that the facts do not fit and telling a rather 
different story. 
A journalist is, of course, generally an ‘outsider’. Investigative devel- 
opment managers may also be ‘insiders’ in being part of one of the 
agencies closely involved in a given situation. However, outsider or 
insider, the important thing is not only to be prepared to reconceptualize 
but to be aware of your own assumptions and actively to look out for 
evidence that may contradict them. 

4 This willingness to reconceptualize implies a two-way relationship be- 
tween ideas and evidence. Ideas come from theory, though not necess- 
arily grand theory. In turn different theoretical perspectives relate to 
competing systems of values. Ideas in this sense are what make up your 
conceptualization of a problem or situation. Evidence is in the form of 
data about that problem or situation. Ideas are what lead you to search 
for particular kinds of evidence, but the evidence may also lead you to 
change your ideas, despite the implied challenge to underlying values. 
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5 In addition to confronting ideas with evidence it is also useful to think 
in terms of confronting one set of ideas with another. This can also be 
thought of as pitting one theory against another to see which gives the 
best explanations in the circumstances under investigation. One set of 
ideas will mean searching for certain related kinds of evidence while 
another set of ideas may lead to finding different evidence. 


6 Investigation should be ‘uneasy’. There must always be the possibility 
of finding out what you didn’t know, not just reinforcing prejudices. 
This implies following up odd pieces of data that don’t fit in with your 
prevailing mode of explanation, rather than ignoring them and concen- 
trating only on data that conform to expectations. 


7 itis often necessary to ‘step back’ and find a different way of looking at 
a problem before being able to make progress. This is really another 
way of saying you should be prepared to reconceptualize. However, the 
point is that you cannot be constantly changing your ideas; generally 
speaking, you work on the basis of a particular theory or 
conceptualization for a while. The idea of ‘stepping back’ is to build in 
review points to ensure that you have a chance of looking at things dif- 
ferently before assuming too readily that your ideas are correct. 


8 Finally, communication of results is a necessary part of any investiga- 
tion. There is more to this point than just the observation that research 
results are useless unless made known to others. Communication of re- 
sults should be thought of from the outset as part of the process of inves- 
tigation. This means, for example, that the question of who or what the 
results are for should inform the way in which the research is designed 
and also have a bearing on what is regarded as sufficient evidence. 


A general question about investigation which has only been hinted at so far 
is how the validity of research is to be judged. Some of the language used 
above (‘the image of a detective’, ‘finding that the facts do not fit’, ‘the 
best explanations’) might fit with the notion of a search for objectivity — 
the realist conception that there is an underlying truth which research aims 
progressively to discover by building up knowledge. However, some of the 
other language used (‘a ‘story’ through which to tell the results’, ‘another 
set of ideas may lead to finding different evidence’, ‘who ... the results are 
for should ... have a bearing on what is regarded as sufficient evidence’) 
fits with a quite different notion, the social constructionist conception of 
the possible simultaneous validity of different accounts or constructions of 
human interactions based on different perspectives, interests and values. In 
this notion, what makes a good piece of research is its utility, which is 
necessarily judged from a particular perspective. 


Indeed, there is a tension between these two positions throughout the book. 
The Conclusion returns to it with a fuller discussion, but it is perhaps not a 
question that can be fully resolved. Certainly investigation can never be 
separated from questions of values. 
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5 INVESTIGATION FOR DEVELOPMENT POLICY AND PUBLIC 
ACTION 

This book’s focus on investigative methods is in the context of investigat- 
ing questions of concern for development policy and public action, not just 
for academic research. We noted in Section | that such questions are sub- 
ject to value-based conflicts over what constitute desirable social and de- 
velopment goals. Thus the above tension about how to judge the validity 
and usefulness of a piece of research is particularly acute in the case of 
investigation for policy and action. 


Policy investigation is also something which necessarily takes place in a 
particular institutional context. Investigation is not done for its own sake, 
but with the aim of informing or influencing the actions of one’s own or 
another agency. Here are some additional points about investigation in this 
context that also form part of the book’s approach. 


1 Conceptualization should include a consideration of agency. This means 
how things are made to happen as well as which agencies are involved. 
It is not just a matter of who has commissioned the research and who 
needs to know the results and to take part in implementation. Questions 
of who is concerned, which organizations and groups are engaged in 
doing what, which interests are implicated by different policy options, 
and what values underlie them, should be part and parcel of how you 
conceive of a development problem or issue throughout an investi- 
gation. 

2 The institutional setting can be a trap. One can be caught in a particular 
(often narrow) way of thinking which may be strongly related to the 
perspectives and values of the particular institutions which have set up 
the investigation in this way. The nature of a trap is that you cannot find 
the way out from inside, so it is hard to say what you should be looking 
for in order to escape. However, it can help a lot to make a point of 
soliciting views from sources outside your agency and those with which 
you habitually work, or to look for literature written from different points 
of view or data collected for different purposes. 

3 ‘Stepping back’ and reconceptualizing may endanger one’s position even 
while being essential to rigorous investigation. Looking at things from 
different perspectives can cause problems at all kinds of levels. It might 
imply questioning your personal values and perhaps uncovering hidden 
biases and prejudices. It might create more overt political problems, not 
only within your agency by questioning the basis of its policies, but in 
terms of relationships to other agencies with important positions in the 
situation under investigation. 

4 The character of available data reflects the previous history of public 
action in a society. In other words, previous policies in the public arena 
determine what data were routinely collected. If anew agency becomes 
involved with a different perspective, or if you changes your ideas on 
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an issue, it is likely that information relevant to the new way of think- 
ing will not be so readily available. This should not in itself be a reason 
for eschewing new ideas, though it might limit the extent of a new 
investigation that can be carried out in a limited time. However, 
the very fact of the limitations of data and the reasons for those limi- 
tations may in themselves be evidence to be fed back into the overall 
picture. 

5 Investigations can themselves be interventions. It is important to con- 
sider not only the cases where investigation informs or influences inter- 
vention by a powerful agency into a situation. There is also the case 
where the actual carrying out of an investigation directly affects the 
situation and should be considered among the interrelated actions of a 
number of agencies in an ongoing process of change. Arguably the lat- 
ter is always the case to some extent. 

6 Communicating results involves questions of rights and responsibili- 
ties. When investigations are aimed at public action they are intended 
to lead to changes which will affect those whose situation was investi- 
gated. Hence communicating the results involves questions of who 
should be involved, who has a right to the information, who will be 
responsible for acting on it, and so on. As noted above, this is not a 
consideration only for the end of an investigation but should inform the 
way an issue is conceptualized throughout. 


The tension between different ways of judging policy-oriented investiga- 
tions suggests that it seems vain to search for complete objectivity, particu- 
larly when the subject of investigation includes policies which may be based 
on competing values. However, it seems dangerous to judge investigations 
simply by reference to utility. Is research successful because it has helped 
the implementation of certain policies or halted others? Whose utility, whose 
interests, are to take precedence? How in that case can one avoid bias in 
the direction of those, necessarily powerful, who are able to commission 
research? 


Difficulties like these have led to the centrality of the idea of ‘challenge’ in 
the book. Whichever view is taken, it seems important for ideas to be open 
— or for biases to be acknowledged — so that they can be challenged as 
rigorously as possible. It may not be feasible to achieve objectivity or to 
agree on the perspective from which to judge utility, but a challenging 
form of investigation should at least avoid incoherent explanations or ad- 
vocating policies which fail to achieve their own aims. 


6 ORGANIZATION OF THE BOOK 

It is the relationship of investigative methods to the policy and public ac- 
tion context which is intended to make this book more than just another 
research methods book. 


INTRODUCTION 


The central part of the book is organized in terms of investigative methods. 
These are designed to cover the main generic skills required for carrying 
out an investigation. Three general methods each have a section: ‘thinking 
with paper’; ‘thinking with people and organizations’; and ‘thinking with 
data’. These central sections on skills or methods follow an opening sec- 
tion on the conceptualization of investigations in relation to policy ques- 
tions. 


Thus the sections are as follows: 


1 CONCEPTUALIZING POLICY-RELATED INVESTIGATION 


The first section looks at how areas for investigation which relate to ques- 
tions of policy and public action can be conceptualized. The first chapter 
uses examples of policy change to explore how different policy contexts 
make it appropriate to formulate different kinds of research questions and 
use different methods. The second discusses boundaries and boundary- 
setting as an important part of conceptualizing areas for investigation, The 
third uses the analogy of how a journalist formulates a ‘story’ to consider 
how the need for results to make an impact affects how an issue is concep- 
tualized from the beginning. 


It THINKING WITH PAPER 

Literature study is to be treated as an investigative method in its own right, 
not just a preliminary. A general chapter on how to ‘do’ a literature study is 
followed by one on the interpretation of institutional discourses as evi- 
denced in reports, publicity materials and so on. 


Ill THINKING WITH PEOPLE AND ORGANIZATIONS 


‘Fieldwork’ includes a particularly enormous range of specific methods, 
including RRA, PRA, organizational assessment, semi-structured inter- 
views, the use of oral testimony, and so on. Hence this section is the one 
with the most chapters. The title “Thinking with people and organizations’ 
indicates that carrying out investigations using evidence gained from in- 
teracting with people at first hand is always a two-way process, whether or 
not a particular method is labelled ‘participative’. 


Development managers may need different sorts of skills to think with 
people and organizations in different situations and contexts, even if they 
don’t always do the fieldwork themselves. Hence, this section will cover 
generic skills at three levels: an appreciation of what is involved in good 
research design and particularly interviewing, both with individuals and 
groups; the development of judgement and an appreciation of rigour in 
order to commission, interpret and evaluate fieldwork carried out by 
others; and an understanding of the relationship between investigations 
about people and organizations and the way results of investigations 
involve and implicate the same people and organizations. 
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Of the chapters in this section, Chapter 6 compares the two main methods 
of using people as informants, namely structured survey and semi-struc- 
tured interview, discusses how rigour can be achieved in each case and 
introduces some concerns about research which is specifically labelled 
‘participative’. Chapter 7 follows this with a more detailed account of 
‘participative’ research methods and considers both the potential for such 
methods to be empowering and their limitations. Chapter 8 looks at inves- 
tigating organizations and their ‘footprints’, to inform either change in the 
organization itself or public action in an arena where the ‘footprints’ are 
found. Chapter 9 gives a personal reflection on some practical and ethical 
difficulties in a case of researching poverty. Finally Chapter 10 considers 
how the need for communicating results throughout an investigation af- 
fects the investigator’s ‘thinking with people and organizations’ from the 
beginning. 


IV THINKING WITH DATA 

The title of this section echoes the previous two to indicate that in all cases 
the concepts brought to the investigation challenge and are challenged by 
the evidence, whether the evidence comes direct from people or from data. 
Chapter 11 covers the analysis of secondary quantitative data, developing 
the basic skill of numerical literacy. Chapter 12 on the analysis of institu- 
tional accounts is to be read in conjunction with Chapter 8 on institutional 
footprints. Chapter 13 deals with critical issues in the use of data, particu- 
larly the case where research results are to be used for advocacy and the 
particularly stringent requirements for rigour which this places on the in- 
vestigation. Chapter 14 applies the concept of challenge to the use of case 
studies which combine quantitative and qualitative data as different forms 
of evidence. 


Finally there is a Conclusion which discusses questions of personal effec- 
tiveness and integrity. It points out some of the differences between the 
chapter authors, and returns to the questions about how to judge the worth 
of a policy investigation introduced briefly in Section 4 of this Introduc- 
tion, before concluding by emphasizing yet again the importance of the 
idea of challenge and ‘trying not to get it wrong’. 
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1 INFORMATION NEEDS AND 


POLICY CHANGE 
STEPHEN POTTER AND RAMYA SUBRAHMANIAN 


It is a very simple idea that different types of policy measures require dif- 
ferent types of information and different skills to gather this information. 
If a policy of water supply involves building dams and reservoirs, then to 
do this requires hydrologists and engineers, with a policy administrative 
system that provides the resources to implement their work. Their infor- 
mation requirements will include measurements of water flow, pressures, 
and dimensions of structures, and they will ask questions about the 
capacity, cost, safety, etc. of alternative configurations. But if the policy 
requires the elimination of leaks from pipes in cities, then different 
engineering skills and information are required. No knowledge or infor- 
mation on dams or reservoirs is needed. Instead, engineers who are skilled 
in leak detection and measurement, technologies to repair pipes in situ and 
systems of water distribution management and monitoring are called for. 


Behind this is a general point about research, which applies particularly 
strongly to policy-oriented investigation. Different policies require differ- 
ent research questions to be asked in order to obtain results that will use- 
fully inform those policies. When the policy changes, the core question at 
the centre of related policy investigations also changes and it is frequently 
necessary to change the method by which those investigations are carried 
out. Furthermore, there can also be a need to change the sort of information 
gathered. 


The first section of this chapter, particularly Figure 1.1, Table 1.1 and Fig- 
ure 1.2, explores this relationship between the nature of an investigation 
and consequent information needs. This section is developed from mate- 
rial used in research training courses at the UK Open and Brunel Univer- 
sities by a team including the first author of this chapter. 


In the following two sections we present two case studies of situations 
where a change in policy has led to a need for changes in the sorts of 
understanding, skills and types of information required. They are inten- 
tionally contrasting. Lessons are then drawn in the final section from what 
they have in common as well as from the contrasts. 


CONCEPTUALIZING POLICY-RELATED INVESTIGATION 


1.1 CHANGED QUESTION: CHANGED INVESTIGATION 


Figure 1.1 is intended to help students or policy investigators focus upon a 
research topic and project aim. Once a project aim is established, a se- 
quence should be initiated that structures how the research is carried out. 
The aim of a research project can often be articulated in terms of a key 
question: for example, “What role could renewable energy technologies 
play in developing economies?’ 


The formulation of a focused project aim then leads on to developing ques- 
tions that need to be answered to achieve the overall project aim. In the 
above example this is likely to involve specifying the type of renewable 


policy area 


research topic 
= 
a ; o 
specific project > 
aim 2 
= 
i 
questions to a) 
address aim 3 
a 
© literature-based 
* secondary analysis of existing data 
© survey 
* semi-structured interviews 
* case studies 
* participative observation 
* action research 
© etc. 


FIGURE 1.1 Sequence for focusing a research project 
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energy technologies considered and the functions they would serve in de- 
veloping economies. From such a consideration should emerge a clear idea 
of the sort of information that is needed to answer these questions. This 
then raises the issue of how this information can be obtained. 


For clarity, the sequence is presented linearly in Figure 1.1, but in practice 
the situation is always more iterative. Obtaining information and under- 
standing may lead to a revision of the project aim or constraints on infor- 
mation availability could lead to a less ambitious project or a different 
methodology. 


It is important to distinguish different types of research question. One use- 
ful way is to think in terms of three dimensions on which to differentiate 
between types of question: 


1 Is the question about the field of policy (including those affected by 
policy or for whom policy is made) or is it a question about policy 
itself? 

2 Is the question a normative or an evaluative one (‘How should...?’; 
‘What is the best way to...?’; ‘How well does X compare with Y?’) or 
is it a descriptive or an explanatory one (about what is happening and 
how to explain it)? 

3 Particularly in the case of descriptive/explanatory questions about the 
field of policy, is it a ‘What?’ question, a ‘How?’ question, a ‘Why?’ 
question or a ‘What if?’ question? 


The first two of these dimensions, taken together, give rise to a matrix of 
four types of question, examples of which are shown in Table 1.1. 


In the straightforward case where those undertaking an investigation are 
able to implement policy themselves or report to those with the power to 
do so, it is questions of the type in the top left box of Table 1.1 that need 
answering in order to decide directly which policies to implement. 
Answering such questions directly (‘How well does such-and-such a policy 
work compared with the alternatives?’ , “What is the best education policy?’, 
etc.) requires a mixture of evidence and subjective judgement. 


In order for a policy investigation to help a decision-maker or develop- 
ment manager to answer such a question, it is often helpful to separate 
out the evidence from the judgement. Gathering evidence generally in- 
volves asking questions aimed at describing or explaining what happens in 
the field of policy, i.e. questions of the type in the bottom right box of 
Table 1.1. 


Although the other two boxes of Table 1.1 do not come into this straight- 
forward picture, they can also provide important aspects of a broader 
investigation into policy. In the bottom left box, questions ask directly what 
people should do and are close to the “What if?’ questions in the bottom 
right box. In the top right box the questions treat policy almost as an object 
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TABLE 1.1 EXAMPL 


Normative/evaluative Descriptive/explanatory 


FOUR TYPES OF RESEARCH QUESTION 


questions questions 
Questions about Should this policy be Which agencies are 
policy itself adopted? stakeholders in this policy? 
How well does policy P How did policy Q come 
work? about? 
What is the best transport Is there the capacity to 
policy? implement the policy? 
Questions about How should people react What do people grow? 
the field of policy to this new policy? How ié cultivati ized? 
What is the best way to live Why do people grow the 
in given circumstances? particular crops they do? 
What if more water were 
available? 


itself, tracing its context, its history, the interests and organizations that 
shape it, and so on. At the very least, investigating questions in this box 
gives a broad background and helps to give a better understanding of the 
policy . 


In practice, the ‘field’ of policy includes beneficiaries who do not simply 
react to the impact of policy or generate demands as predictable responses 
to certain causal factors, but act independently and indeed implement ‘poli- 
cies’ of their own. The strong movement towards different forms of 
participative research is one recognition of this. Whether research is car- 
ried out in an overtly participative way or not, it has to be recognized that 
the boundary between policy (and those agencies that make policy) and 
the field of policy is at best fuzzy and at worst a construct which may 
hinder rather than help investigation. 


Nevertheless much research undertaken or commissioned by development 
managers is aimed at gathering evidence about what we have called the 
field of policy. This can often be quite similar to research undertaken by 
postgraduate students, many of whose topics would not have been spon- 
sored without some policy relevance envisaged for the results. We have 
already noted how research questions about the field of policy differ con- 
siderably amongst themselves according to whether they begin with the 
words ‘What’, ‘How’, ‘Why’ or ‘What if’. These differences reflect whether 
the investigation’s core interest concerns: 


(a) description and basic data gathering (typically involving questions be- 
ginning with the word ‘What’ or the phrase ‘How many’ (for example: 
What is the pattern of energy demand in a particular country?); 
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(b) an analysis of how something happens (for example, explaining how 
electricity demand arises from the domestic, commercial and indus- 
trial sectors); 


(c) an understanding of causality (for example, the nature of the relation- 
ship between electricity demand and economic activity in the com- 
mercial sector); 


(d) being able to predict what will happen if causal factors change (for 
example, if electricity tariffs were altered, how would the pattern of 
demand alter?). 


These categories are, of course, somewhat artificial and, in practice, a project 
may encompass a range across this spectrum. But this is a useful way of 
thinking about how deep an understanding is needed of what causes an end 
result, and if an important part of the project is to intervene in the system to 
change that end result. As an investigation shifts across the spectrum from 
descriptive ‘What?’ and ‘How?’ questions to contemplate the ‘Why?’ and 
“What if?’ questions, the sort of information that is needed changes and the 
methods of investigation change as well. Broadly, this shift involves a 
change from the gathering of basic factual information and ‘hard’ statisti- 
cal methodologies through to information and methods that allow an un- 
derstanding of systems and involve ‘softer’ qualitative information as well 
as modelling methods. This is broadly summarized in Figure 1.2. 


* experiment 


* scenarios 
© multivariable modelling 


* qualitative interviews/panels 


* case studies 


* semi-structured interviews 
z * surveys 
How? * simple modelling 


© semi-structured interviews 


* surveys 


‘administrative’ statistics 


FIGURE 1.2 Descriptive and explanatory research questions and methodologies 
of investigation 
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These considerations of identifying the nature of the questions being asked 
and what sorts of information and methods of investigation are needed 
apply as much to policy and public action as they do to a postgraduate 
student’s project. Indeed, even though research students may take some 
convincing on the matter, this is more difficult than in a PhD or Masters 
project. A student can relatively easily shift to a different methodology, but 
entire professions with specialist skills and methodologies have built up 
around key areas of policy-making. Departmental structures have been 
assembled around these methodologies, skills and professions. So what 
happens when there is a shift in policy that requires new or different skills? 
What happens if a policy shifts to become the domain of a new govern- 
ment department or, horror upon horrors, requires close co-ordination be- 
tween previously separate departmental empires? Do institutional struc- 
tures represent a significant barrier to developing new policy directions? Is 
the problem even acknowledged? 


Increasingly, public policy is requiring a shift across the investigative spec- 
trum towards a deeper understanding of systems, rather than taking the 
systems for granted and responding to their outputs. This shift arises for a 
number of reasons, not least the increasing uncertainty in which policy- 
making takes place. A common type of change, explored in the two case 
studies in this chapter, is from a relatively stable policy environment of 
reacting to demand and administering established public services, towards 
a developmental policy involving active intervention and trying to manage 
demand in order to achieve development goals. The consequences of this 
shift in the level of understanding needed are rarely appreciated. If a stu- 
dent submitted a thesis claiming to understand how a system would react 
to possible external changes without using a research method that pro- 
vided an understanding of that system, the chances are he or she would 
fail. Yet today, in many policy areas, initiatives to influence and change 
systems are under way, serviced by only the barest information of the ‘How 
much?’ variety. It is just as well that such policy programmes do not have 
to be submitted to a university examination panel! 


1.2 UK TRANSPORT POLICY 


Recent changes in UK transport policy provide a good example of research 
method and data implications when there is a change in the core policy 
rationale. 


Since the Second World War, the traditional approach to transport plan- 
ning in Britain, and most nations, has been an essentially reactive one, 
with the state responding to changing travel demands. Predominantly this 
has involved providing increased road capacity, together with the progres- 
sive trimming of rail and bus services, with some exceptions due to ‘social 
need’ (e.g. subsidizing local bus, rail and ferry services, particularly in 
rural areas). The way in which these services have been provided, particu- 
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larly whether by the state or the private sector, has been the subject of 
numerous reforms. These have culminated in the privatization in the 1980s 
and 1990s of bus and rail services. However, these reforms are within the 
context of a reactive policy ethos that has taken changes in travel demand 
for granted. 


Such a reactive policy response can be serviced by relatively straightfor- 
ward information. Basically this requires information on demand for travel 
infrastructure and services (“What?’ or ‘How much?’), with little need for 
any understanding about how this demand arises. Demand is taken for 
granted and the policy response is to accommodate it. 


Only in large cities, particularly London, has this reactive approach been 
rendered impossible by the sheer volume of traffic concentrated into a rela- 
tively small area. Even so, the general desire has been to provide as much 
road capacity as possible. 


This traditional ‘demand-led’ transport policy has now lost its credibility. 
By the early 1990s, transport was recognized as a key environmental prob- 
lem area. Ata local level, despite the application of ‘cleanup’ technologies, 
transport emissions remain a serious air quality and health threat. Less 
amenable to technological fixes are transport’s emissions of carbon diox- 
ide (CO,), the key gas contributing to global warming. 


The UK, along with most developed nations, has an international obliga- 
tion, under the 1992 Climate Change Treaty, to stabilize CO, emissions at 
1990 levels by 2000 with the prospect of further treaty obligations requir- 
ing a cut thereafter. The transport sector is the fastest-growing source of 
CO, emissions. CO, emissions from transport have risen from 22 MtC 
(million tonnes of carbon) in 1970 to 37 MtC in 1990 and, even allowing 
for current policy measures, are forecast to rise to 42 MtC by 2000. 


In the short term, a decline of CO, emissions for electricity production 
associated with the shift from coal-burning to gas-burning power stations 
has let transport off the hook. But this only delays the eventual need to 
address transport’s growing global environmental impacts. This is re- 
inforced by calls for substantial cuts in CO, emissions by the early years of 
the twenty-first century and scientific evidence suggesting that a cut of 
60% or more is needed simply to contain global warming to an acceptable 
level. 


However, although recent problems concerning the local and global envi- 
ronmental impacts of motor traffic are growing in importance, a more im- 
mediate factor for the re-evaluation of Britain’s road-building policy was 
the inability of road-building to reduce traffic congestion. It was the publi- 
cation, in 1989, of government road traffic forecasts that brought home the 
reality that accommodating predicted traffic growth was impossible. The 
1991 update of the road traffic forecasts predicted an increase of between 
72% and 121% in road traffic over the period 1990 to 2025. 
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In 1991, a key piece of research, Transport: the new realism (Goodwin et 
al., 1991), showed there is no way that Britain could physically, economi- 
cally or socially accomodate the Department of Transport’s forecasts for a 
100% or more increase in traffic. In 1992, the Institution of Highways and 
Transportation noted that road congestion is likely ‘to occur in more areas 
for longer periods of the day,’ with ‘an increasing likelihood of urban 
gridlock’ (IHT, 1992, p. 5). 


Since the early 1990s, there has been a gradual acceptance that, rather than 
simply building new roads and widening existing ones, policies to manage 
the demand for motorized travel will be needed in all towns and cities. 
Demand needs to be shifted away from cars to public transport, walking 
and cycling. However, for long-distance, inter-urban traffic, the concept of 
demand-led investment remained intact, if under siege. Thus in the early 
1990s, specific proposals were developed by the Department of Transport 
for widening several motorways and other major roads, including capac- 
ity-enlarging bypasses on strategic trunk roads. At a European level, the 
Transeuropean network motorway programme is also rooted in such an 
ethos. 


The cost of Britain’s capacity expansion programme was considerable and 
in late 1993 the Department of Transport announced that it had risen to £5 
billion. However, in the Budgets of 1994 and 1995 the road-building pro- 
gramme was severely cut and many of the road capacity enlargement 
schemes deferred or abandoned. To a large extent this apparent ending of 
Britain’s road-building programme arose through Treasury opportunism; 
there was a need for substantial public expenditure cuts and the roads pro- 
gramme was a ‘soft target’. That the roads programme was seen as such by 
the Treasury is perhaps the more significant point. In the past it had been 
heavily protected. 


A number of significant government-commissioned reports in 1994 left the 
rationale for a major road-building/upgrading programme in tatters. The 
Royal Commission on Environmental Pollution’s transport report indicated 
the need for radical measures to manage the demand for transport if envi- 
ronmental objectives were to be taken seriously. This was closely followed 
by the 1994 SACTRA (Standing Advisory Committee on Trunk Road As- 
sessment) report on the methods to justify and assess new road schemes, 
indicating the role that new roads play in actually generating more traffic, 
which led to serious doubts over the way roads are financially justified. 


This situation was summarized by Goodwin (1994), who argued that trans- 
port policy’s ‘New Realism’ is entering a second phase. The first phase was 
the gradual (if grudging) acceptance that for major towns and cities 
demand management has to be the core rationale for transport policy. The 
second phase involves the realization that demand restraint is inevitable 
for trunk roads and motorways also. Quoting British Road Federation 
research, Goodwin showed that even a road-building programme of an 
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uncontemplated vastness (beyond that Britain’s economy could sustain) 
would fail to stop congestion getting worse. As a policy response, road- 
building will always fail; demand management is the only direction possi- 
ble for transport policy at all levels. This may be increasingly accepted at 
an intellectual level, but is only just beginning to seep into the culture of 
policy-making and government. By the mid 1990s, the language of trans- 
port policy-makers had changed significantly, but reality was slow to fol- 
low the rhetoric. This was epitomized by the 1996 Green Paper on trans- 
port, The Way Ahead, which summarized well that continued traffic growth 
was impossible to accommodate, but provided little by way of policy meas- 
ures that would achieve an alternative vision. Just admitting that the old 
way fails to work cannot of itself produce a new way that does. 


Developing policies that are appropriate for an increasingly uncertain fu- 
ture requires considerable understanding of how demand for travel arises. 
This involves answering explanatory questions about the field of policy 
(the bottom right box in Table 1.1). With such an understanding uncer- 
tainty could be more easily accommodated. There are strong structural 
changes in British society and economy that have led to the historical rise 
in mobility, and traffic in particular. It is a self-reinforcing process that has 
for several decades increased the amount of travel that people undertake. 
This is a process that is far from complete, as projections for future levels 
of traffic and travel indicate. How could policies influence these structural 
causation factors that for so long have been taken for granted? 


1.2.1 INFORMATION NEEDS FOR DEMAND-MANAGEMENT TRANSPORT POLICIES 
Understanding the key factors behind the generation of demand for travel 
becomes increasingly important when the question of planning for future 
accessibility and mobility arises — in other words, when transport policies 
shift from just reacting to demand to aiming at developmental goals. 


There is now a need both to manage traffic demands (e.g. vehicle occu- 
pancy, method of transport used) and the factors that affect the generation 
of these demands (e.g. land use planning — such as where the workplace, 
shops and social facilities are, compared with where people live). 


Managing the demand for traffic growth requires an understanding of what 
are the key components of change and the underlying causal factors. To 
put this in terms of types of research questions, in order to answer the 
normative policy question ‘What is the best way to manage traffic demand 
in order to achieve goals of accessibility and mobility?’ we need first to 
ask some descriptive and explanatory questions about the field of policy, 
i.e. the traffic itself: ‘How and why does demand for traffic grow?’. 
Understanding causal factors would permit an identification of where trans- 
port demand management policies can be most effectively applied. With- 
out such an understanding, policy will be merely tinkering with symptoms 
and not causes. Equally it is important to identify where policy can and 
cannot make a difference — i.e. what traffic-generating elements need to be 
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accepted and which are amenable to policy influence. It is also important 
to identify where other areas of policy have significant indirect transport 
effects (e.g. changes in schools admission procedures or patterns of health 
care leading to changed transport demands). It is now accepted that plan- 
ning and land use policies need closer integration with transport policies, 
but the boundary has not extended to include other key policy areas such 
as health, education, welfare or fiscal policies. 


Ideally, in this case, ‘finding out fast’ would imply having new generic 
information to hand that can provide quick answers to changing policy 
needs. However, managing transport demands is a new and very uncertain 
field, and comprehensive information on the nature of demand is not avail- 
able. The information needs of traditional, demand-led, transport policies 
were relatively simple, involving statistics measuring demand, for exam- 
ple, traffic numbers, flows, origins and destinations, vehicle types etc. How 
this demand was generated was not of particular importance. Thus before 
the policy change, finding out fast meant looking up and interpreting data 
supplied by established information systems. It also means recognizing 
the limitations of such sources. 


Policies seeking to manage traffic growth require the development of a 
package of co-ordinated policy measures and integration with other areas 
of urban policy intervention (e.g. land use planning, employment policies, 
health, education and welfare provision). Once policy shifts from respond- 
ing to demand to seeking to influence demand, there is a need for infor- 
mation to understand causes not merely statistics of measurement. Rather 
than seeking factual information on ‘How much?’, transport planning now 
needs to add answers to questions beginning with the words ‘How’ and 
‘Why’ and, increasingly, to contemplate “What if?’ (e.g. How does de- 
mand arise, why does it arise and what if changes are made?). Finding out 
fast now implies special investigation on particular topics, since the 
standard statistics are in the wrong form for answering the new questions, 


As noted at the beginning of this chapter, as you proceed across the inves- 
tigative spectrum there is a shift away from hard ‘scientific’ type approaches 
towards softer, more socially based methodologies. This requires the use 
of techniques and measurements that have previously been on the fringes 
of transport policies. These are behavioural methodologies that can assess 
the public acceptance of demand management measures and the effects of 
different combinations of measures, over time and in different places and 
circumstances. Equally there is a need for the transport modelling method- 
ologies that have emerged to be modified, at the same time as a more com- 
plete and dynamic understanding of travel and traffic generation is sought. 


This sort of understanding will have to be rapidly assimilated if traffic 
growth demand management is to have any chance of success. Yet the 
Department of Transport’s survey of data routinely collected by local au- 
thorities (TechEcon, 1995) indicates the existence of virtually no infor- 
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mation other than that measuring travel demand. In other words the new 
demand management approach is being serviced by the information sys- 
tem of yesterday’s policies. 


The report on extant data in local authorities documents the existence of a 
large amount of factual information, including traffic counts, traffic speed 
surveys, parking supply figures, public transport supply, and accidents. For 
historic reasons, local authorities do have good land use data and a min- 
ority have conducted attitudinal and behavioural surveys that could have a 
relevance for managing traffic demand. There is no indication of the exist- 
ence of local household travel surveys, the first step in the rung to analys- 
ing local travel behaviour. 


In recent years, transport policy has moved from responding to demand for 
travel, to analysing how demand arises and seeking to manage it to fulfil a 
set of developmental policy goals. This shift requires additional skills, un- 
derstanding and eventually additional information systems. In the mean- 
time, policy investigations have to be undertaken to answer new questions, 
even though the statistics available come from the old systems. This can 
relate to a danger of finding out fast, i.e. using information that is there 
rather than what is suitable. 


1.3 POLICY SHIFTS IN PRIMARY EDUCATION IN INDIA 

The case of transport policy in Britain illustrates the complexity of re- 
lationships between the institutions which make public policy, and the peo- 
ple for whom policy is designed. In particular, it highlights the extent to 
which planning for public policy need not be just an issue of reaction to 
expressed demands, but can involve intervening in order to manage de- 
mand to try to achieve developmental goals. This involves understanding 
the dynamics of ‘demand’ and its causes, in terms of the changing needs 
and interests of users. This is true of most public service sectors across 
different national contexts. To illustrate this point we will be looking at 
education policy in India, and assessing the shift in information required, 
prompted by changing directions in policy goals. 


Changes in policy take multiple forms, arise from a variety of contexts, and 
are influenced by different actors. Often policy changes are evolutionary, 
and involve making shifts and adjustments rather than effecting disrup- 
tions with the past. These shifts could be in terms of restructuring a particu- 
lar intervention in a sector to include new constituencies, and/or putting 
into place new mechanisms for intervention. The National Policy on Edu- 
cation (NPE), introduced in India in 1986, is an example of an evolutionary 
policy shift triggered by several factors: a change in political leadership, 
bringing with it a renewed focus on the importance of education for mod- 
ernization and development; a recognition within government that national 
education achievements have not matched projected goals over years of 
planning; a renewed international focus on education, particularly primary 
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education and women’s education, spearheaded by the World Bank and 
strongly influenced by its ideological framework and financial support; and 
finally, the activism of social movements including the anti-child labour 
campaigns, literacy campaigns and organizations of the women’s move- 
ment. The interplay of these different factors over a period of time saw the 
birth of a new policy approach to education, which was built on the lessons 
of the past, but with elements that emphasized new ways of doing things. 


The NPE was produced through a lengthy consultative period, and based 
on a comprehensive sector review prepared by the Department of Edu- 
cation. The review identified several symptoms of systemic failure: low 
enrolment rates, with disparities across social groups based on gender, caste, 
ethnicity and geographical location; high discontinuation rates; low 
academic achievement rates; and overall high rates of illiteracy. These 
symptoms were exacerbated by the low levels of resource allocation, aver- 
aging 3% of the GNP, with declining expenditure on primary education, 
insufficient infrastructure to meet the needs of sections of the population, 
particularly in rural areas, high rates of teacher absenteeism, and overall 
poor management of schools, particularly in terms of monitoring and eval- 
uation. The review, published in 1985, was followed by the NPE in 1986, 
and then by a Programme of Action in 1992 and a revised policy statement 
on Education for All, in 1993. These documents present a consolidated 
policy discourse on education, which represent some significant changes 
in the approach to education planning. 


Particularly significant shifts are the explicit commitment to equality, em- 
powerment and a proactive role for government. The NPE accords priority 
to the ‘removal of disparities and equalising educational opportunity by 
attending to specific needs of those who have been denied equality so far’, 
with a particular emphasis on using education to empower women. The 
NPE also envisages a ‘positive interventionist role’ for the education sys- 
tem, viewing it as a vehicle through which the government can act as a 
catalyst for social change. Given the entrenched social and economic in- 
equality in India, it is a significant policy commitment. The innovation or 
shift in policy discourse can be identified both in the explicitness of these 
statements and in their location as a central statement of public policy goals. 


In this section we look at the types of information needed, the mode of 
investigation required, and the research questions that require answers, in 
order for this new policy approach to be effective. 


At the broadest level, we can ask normative or evaluative questions about 
the changing policy itself in its context. For example: How should more 
specific programmes be developed from the new policy approach? Is the 
new approach ‘better’ than the old? 


Before thinking about research questions, however, we should check there 
really is something substantial to investigate. To what extent can altera- 
tions in policy language be considered to constitute a policy shift? In as- 
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sessing the evidence here, there are two points that need to be borne in 
mind (which apply to the transport policy case study as well). 


1 Policy documents are general statements of intention, and do not serve 
as a guarantee of implementation or action, even where they use pro- 
gressive terminology and promise radical change. On their own, they 
tell us little about how changes will be put into practice, and provide 
little indication of the ‘real’ nature of the change that is likely to be 
experienced. For example, using intangible concepts like ‘equality’ and 
“empowerment to describe policy goals may provide an important con- 
ceptual framework, but these terms are likely to remain relatively mean- 
ingless unless they inform the design of programmes, and become val- 
ues shared by policy actors including planners, programme implemen- 
tors and users. In the history of education planning in India, there have 
been numerous commissions, reviews and reports identifying most of 
the problems cited in the NPE policy documents. The universalization 
of education has always been the priority or goal cited in policy docu- 
ments, including those of the five-year plans produced by the country’s 
centralized Planning Commission, The NPE documents provide little 
insight into why, despite earlier policies, outcomes have not matched 
expectations or goals. 


2 Policy documents are often political documents, but disguise from their 
public the politics of the decision-making process that precedes and 
influences their formulation. Policy documents often use neutral termi- 
nology that does not indicate the nature of the decision-making proc- 
ess, or illuminate the reasons why some strategies are included and not 
others. While experienced observers of policy processes may be able to 
identify the critical influences on policy options (either in terms of in- 
fluential individuals such as politicians, or agencies like the World Bank), 
the general public would have little insight into the politics of policy 
formulation. 


Because policies tend to represent a generalized outline of desired ‘ends’, 
research that seeks to evaluate the impact and/or inform the direction of 
policy change needs to be based on more concrete events in the policy 
process. Grindle (1980) identifies implementation as the critical stage where 
public policy is executed after the goals and objectives have been designed, 
and resources are allocated. Specific programmes then become the means 
through which policy goals are delivered. Research focusing on programmes 
can often provide insightful snapshots of policy, capturing the dynamics of 
policy change at the meso or micro levels, and demonstrating how categ- 
ories of analysis developed on paper are converted into action. In the case 
of education policy in India, new programmes have been designed and 
established in selected districts of the country; they highlight the shifts 
made in the NPE from merely addressing shortfalls in education provision 
to creating a climate for innovation. These programmes focus on both adult 
and primary education, and two are discussed in a little detail below. 
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1.3.1 INNOVATIVE PROGRAMMES IN INDIA 

The Mahila Samakhya programme, launched in 1989 by the Government 
of India with donor assistance, was born out of the consultative process of 
the NPE. Aimed at adult women in poor communities in selected states 
and districts in India, the programme is based on a reconceptualization of 
education ‘as a process of ongoing collective action and reflection which 
would empower women’ (Government of India, 1995), using process-based 
methods of raising consciousness and awareness. Education, in the formal 
sense of literacy and numeracy, is not imposed on the target group, women. 
The principles of the programme stress that the pace and the priorities of 
the programme are to be determined by the women and communicated 
through their facilitators to the programme offices at district and state level. 
The programme is based on a reversal of traditional programme hierarchy, 
empowering the users to make decisions, while programme staff are viewed 
as facilitators. Significantly, Mahila Samakhya was not designed as a re- 
source-intensive programme, but focuses on sustainability through build- 
ing people’s organizations at the village and community levels. 


The District Primary Education Programme (DPEP), officially launched 
in 1992 with principal financial assistance coming from the World Bank, is 
aimed at strengthening the base of the education system, the primary levels 
of schooling. The DPEP strategy is the restructuring of the primary edu- 
cation system in selected districts, and has as its particular focus the 
securing of education opportunities for underprivileged social groups. The 
programme has ‘hard’ targets, in terms of improvements in enrolments, 
retention and achievement within a time-frame, and has resources allo- 
cated for infrastructure development, such as building more schools and 
improving water and sanitation provision in existing schools. However, 
within the resource and target framework, DPEP emphasizes the use of 
mobilization campaigns, and builds links between government agencies 
and community bodies empowered to make recommendations to govern- 
ment about aspects of the programme. DPEP also focuses on retraining 
teachers in terms of both curriculum changes as well as the more process- 
based aspects of the programme. 


Mahila Samakhya and DPEP, despite the differences in focus, share sev- 
eral features which provide concrete attempts to put the key policy shifts in 
the NPE into practice. 

* They both specifically target underprivileged and traditionally excluded 
social groups, in terms of mobilizing their participation in education and 
also committing government resources to investment in infrastructure. 

* Both programmes restructure management to make it responsive and 
flexible. This is particularly so in the case of Mahila Samakhya, though 
DPEP also states that ‘capacity-building’ and establishing appropriate 
management structures are important components of programme strat- 
egy. 
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¢ Both programmes build their strategies on a network of community or- 
ganizations and community decision-making as an integral part of their 
programme functioning. 


Each of these aspects reflects the framework of the NPE, and provides a 
crystallized and tangible context within which policy shifts in education 
can be investigated, monitored and evaluated. 


1.3.2 FOUR LEVELS OF POLICY INVESTIGATION 


The case study in Section 1.2 revealed how shifts in transport policy in 
Britain have necessitated shifts in the type of information required to sup- 
port policy structures, Clearly, investigations are relevant to different points 
and levels in the policy process, and can use existing information systems 
in different ways and to greater or lesser extent. Investigations can be de- 
signed to identify the need for change and the content and direction of 
change; or they can be designed in the aftermath of policy change to pro- 
vide an evaluative perspective. If policy is conceptualized as an iterative 
and dynamic process, rather than a linear progression from design to im- 
plementation and then outcomes, it will help to locate information systems 
within the constant process of policy change. Policies are rarely written on 
‘clean sheets’, but are, variously, responses to ‘problems’ that are perceived 
to exist based on evaluation and feedback, or prompted by a sense of ‘cri- 
sis’, or brought about through changing social and political environments. 


Whereas the UK transport policy case was written to reflect the change in 
the type of investigation needed to inform policy-makers, in this Indian 
case we are looking at how the whole policy process may be investigated 
from a more independent viewpoint. From this perspective, we suggest a 
framework for independent policy investigation based on four parameters, 
each of which leads to research at a different level. They are: 


1 the concept underpinning policy change; 

2. the contexts of policy change, in terms of the environment within which 
the concerned policy is designed, as well as the complex of ‘reality’ in 
which it seeks to intervene; 

3 the capacities for change, particularly with regard to the institutions 
involved in implementing the policy change; and 

4 the experiences of change of those who are the intended beneficiaries 
of policy. 

In terms of the distinctions used in the discussion of research questions in 

Section 1.1, the first three levels are about the policy itself (mostly top 

right box of Table 1.1), while the last is about the field of policy (bottom 

right box in Table 1.1). 


Researching concepts 
Locating policy goals and targets within debates about the kind of change 
that is desirable is an important first step in evaluating the viability of policy 
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goals. If policy goals are situated within an ideological perspective that 
does not correspond to the perspectives and needs of a majority of targeted 
users, will it impede the realization of those goals, and if so, to what ex- 
tent? In the case of education planning, there are interesting ideological 
conflicts. Formal education has largely been a twentieth-century concern, 
and discussions of its value and importance continue to be debated. As 
Jones notes, ‘The history of education in the twentieth century has been 
very much concerned with the need to choose between the moral and the 
material purposes of education. Were education’s principal effects the un- 
locking of individual potential or rather the enhancement of social admin- 
istration? Did education foster liberty or stifle it? Did it promote creativity 
and entrepreneurship or rather passivity and subservience? Was it a vehicle 
for the renewal of society or its mere reproduction?’ (Jones, 1992, p. xiii). 
Conflicting perceptions of the importance of education are central to the 
relationship between the system and its users in India, where attendance at 
schools is not compulsory. This makes the policy arena in education a par- 
ticularly contested one, where different actors are likely to define the value 
of education in different terms, and possibly expect different outcomes 
from the system on offer. 


Researching political context 

Beliefs about the importance or direction of policy goals are not the only 

influence on policy design. The chosen boundaries of policies and pro- 

grammes also reflect political decisions on what is viable and what is de- 

sirable. Hence, investigating the political factors that contribute to policy 

shifts as well as the selection of design and strategy is often a critical as- 

pect of research on policy change. At the level of decision-making and 

political authority, questions could include: 

© What were the influences on the policy shift (both individual and insti- 
tutional)? 

* Whatare the roots of the policy shift? (in political ideology? in crisis? in 
responses to evaluation?) 

¢ Is the timing of the introduction of the policy shift politically signifi- 
cant? (election? crisis?) 

* What is the extent of the change desired? How different is the policy 
from previous efforts? What is the history of the ideas involved in the 
change? 


The political context of decision-making is relevant at all levels of the 
policy process, and so investigation involves looking into the contexts of 
power and authority that determine policy design. 


Education policy has the power to shape and transform the lives of those 
who benefit from it in a variety of ways. A commitment to equality would 
necessarily involve transforming existing social relations to remove the 
barriers that exclude disadvantaged sections from schools. Management of 
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the education system in a local context would require information about 
local social relations so that the achievement of policy goals of equality 
and empowerment can be monitored. Thus policy research should include 
identifying the organizations and actors participating in the policy process 
and locating them in a network of relationships, both social and organiza- 
tional. Within this framework of structures and agents, it will then be easier 
to identify which relationships are likely to have which impacts on the 
realization of policy goals, and the direction of change. A suggestive ‘map’ 
of policy actors in the context of an education programme aimed at rural 
areas could look like that in Figure 1.3. 
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FIGURE 1.3 Example of mapping the socio-political context of education policy 
in India 


The ‘map’ in Figure 1.3 is merely a subjective draft, but could be as de- 
tailed as required within a given context. What it does help to do is to 
identify the key actors involved at the different stages of the policy proc- 
ess. A map of this kind is useful in the process of updating the researcher's 
knowledge of the context and can serve as a reminder of the complexity of 
the policy process. 


Researching institutional capacity 

A third level of analysis involves the stage at which policy goals are made 
operational. The implementation stage of the policy process is an integral 
link between policy design and the realization of outcomes. In the two 
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education programmes discussed, the administrative and management struc- 
tures form a critical component of the process of change. Research would 
need to focus on the capacity of implementing agencies to adapt to the 
changes in policy, and locate the analysis within an understanding of these 
agencies as organizations with specific histories and internal relations. To 
what extent are implementing organizations flexible in their structure to 
adapt to changes in practice that policy shifts may necessitate? How re- 
sponsive are they, and do they need to be, to users’ interests around the 
resources that they have the power to disburse? This analysis may be criti- 
cal to understanding the dynamics of policy change at the interface be- 
tween organizations representing the state and wider constituencies repre- 
senting civil society. A representation of this is shown in Figure 1.4. 
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FIGURE 1.4 Organizational context of implementing agencies 


Researching the field of policy 

A final, important level of analysis would need to focus at the level of the 
people whom policy change is expected to benefit. Researching change at 
the level of users is necessary not just to evaluate ‘impact’ in a linear sense, 
but to evaluate the likelihood of policy changes meeting the expectations, 
needs and interests of users. Here it may be important, as in the UK trans- 
port policy example, not only to find out what underlies or causes change 
in expectations or needs, but also to ask “What if?’ questions. If policy 
aims to develop or manage demand, existing expectations may be a poor 
guide. It is also useful to distinguish needs from interests. This kind of 
investigation is particularly vital in a context where policies claim to be 
concerned about the interests of disadvantaged sections of the population 
whose voices may not be heard by planners situated in offices removed 
from the field. However, one should not confuse interventionism with the 
imposition of external values. The experience of users can provide a per- 
spective on policy change that can point to gaps in analysis or information, 
and help to explain why outcomes may not match expectations. 
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Research and investigations on policy change at all these four levels are 
integral to the completion of the loop between the implementation of policy, 
user responses and the design of policy change. If designed and carried out 
by individuals or institutions working outside government, research can 
play an important part in promoting public action and advocacy for change. 
The ideas and evidence that research offers can provide the basic backdrop 
against which policy is put into operation. 


1.4 POLICY CHANGE AND INFORMATION NEEDS 


A number of general points about policy investigation can be made from 
the areas of contrast as well as the common themes emerging from these 
two case studies. 


Implications of change from reactive to developmental policy 

A stable and purely reactive policy regime requires only answers to ‘What?’ 
questions about the field of policy. However, once there is a policy change 
towards demand management or developmental policies there will also be 
a change in the type of question which needs answering. This could be a 
change towards needing to understand causes and the likely impact of new 
policies, i.e. a change from ‘What?’ through ‘How?’ to ‘Why?’ and ‘What 
if?’. It could also be a change towards investigating not just the field of 
policy but also the policy itself and the context of policy change, including 
the multiplicity of stakeholders and policy agencies involved as well as the 
capacities of various agencies for change and the competing concepts be- 
hind changing policy goals. 


Stakeholder involvement 

A shift towards a policy-as-process approach involves a change in the role 
of stakeholders — planners, researchers, activists/advocates and users/ben- 
eficiaries/clients etc. There is a need to involve and empower stakeholders 
in policy change to achieve acceptance and success. In both case studies, 
existing institutional structures have proved a barrier to this happening. 


Role of research 

As noted in the Introduction to the book, investigation also becomes a tool 
or means for policy change through the entire process. This is the case 
whether it is undertaken or commissioned by policy-makers with the direct 
aim of informing policies, or whether it is undertaken from a more inde- 
pendent perspective, when its results might be used for lobbying or advo- 
cacy. 


In the UK transport policy case study, research tended to show that the 
traditional policy approach was not working rather than a new process that 
did work. This gap remains at the time of writing. In the Indian education 
study, investigation of new programmes such as DPEP may aid the im- 
proved design and development of such programmes within the new ap- 
proach. 
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The need for research results to be available throughout the 
intervention 

A policy change could involve the outline of a broader parameter of change, 
while the information gathered can help to develop the direction of change 
in greater detail. Often the research can be the only contact that planners at 
higher levels have with reality; it thus provides not only a symbol of the 
‘public’ for planners, but also their means of defence, justification and le- 
gitimacy. Having research results available throughout an intervention also 
helps them to be open to incorporate feedback, lessons of evaluation and 
so on, to complement the dynamic of the policy process. 


How identification of information needs depends on the nature of 
the policy change 

Policies can be changed in different ways: 

¢ the change could represent a shift in ideology underpinning policy; 


* the change could be an improvement on previous efforts based on feed- 
back or complaints, i.e. lessons from evaluations; 


* the change could be about the recipients of benefits rather than about 
content; 


* the change could be about implementing agencies or management struc- 
tures. 


In both examples here, the change was from a reactive to a developmental 
orientation. In the transport case study, the policy change centred on the 
abandonment of a failed ‘road-building’ approach, and the adoption of the 
idea of managing demand in order to achieve goals such as accessibility. 
This included the realization that traffic growth had to be controlled, but 
with only a vague idea of what was needed to achieve this. In consequence, 
information needs have not been adequately identified. In the Indian edu- 
cation study the change is from administration of schools and other educa- 
tional services towards recognizing the developmental role of education 
and an interventionist stance, targeting certain socio-economic groups and 
reforming management structures. 


The need to identify the sources and contexts of policy change 

This relates to the above point. Policy changes are principally, or ultimately, 
an issue of political will. However, the key catalysts of policy change do 
vary, depending on the sector, the particular policy, the country, political 
structure and so on. 


In the transport case study, lobbyists that had for many years been on an 
uninfluential fringe rapidly became part of the core political process. These 
included the public transport or environmental lobby, such as Transport 
2000 and Greenpeace. They joined the traditionally powerful industry lob- 
bies, such as the British Road Federation and the Confederation of British 
Industries, both of which have changed their stance in response to the fail- 
ure of traditional road-building policies. The extension of political recog- 
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nition to wider stakeholder interests was perhaps epitomized by the award- 
ing in 1996 of an OBE to the Director of Transport 2000, something un- 
thinkable a few years ago. 


In the Indian education study, political context was recognized explicitly 
as one of the levels of investigation required, and Figure 1.3 showed the 
kind of result to be expected, in terms of a ‘map’ of policy actors. 


Context could include both crisis situations and those where changes are 
more gradual. They also include the nature of the political system which 
can determine the boundaries of policy change; the time-frame planned for 
the policy; the capacity of implementing agencies; and so on. 


The framework for designing the investigation will need to be based on 
assessments of the key actors involved in policy change as well as the 
political context which shapes the direction of the change. This will pro- 
vide more ‘realistic’ parameters within which the policy change can be 
investigated or information to feed into it can be collected. 


The scope of policy change 
This is suggested in the earlier points above, but needs to be reiterated. 


The ‘scope’ of change is not just about the influence of political variables 
or institutional agendas on policy change, but also could be more simply 
about the extent to which new institutions are likely to be created, or new 
actors embraced. If the policy change is a shift in the criteria developed for 
building new schools (a simple example), it could range in scope — on the 
one hand, it could be the simple commission of research to identify new 
criteria, retraining for old implementors etc., or on the other, it could be a 
complete change in the institutions that formerly were involved in plan- 
ning, building etc., and the creation of new community institutions in order 
to identify criteria in a participatory way. Identifying the scope of change 
will help to focus the direction of the investigation, and help to ensure that 
information is relevant, presented in a way that is directly useful, and not 
overly ambitious. The ‘scope’ is essentially a combination of assessing the 
nature and content of the direction of policy change and the sources and 
contexts of change. 


The key point here is that investigations are not designed in a vacuum; they 
only exist around a bounded rational entity called ‘policy’, but are rooted 
in several traditions or histories, intellectual, cultural, political and devel- 
opmental. What they offer to planners will depend on the ‘field of vision’ 
of those who carry out the research, what is perceived, understood or inter- 
preted by those who use it, and then what is realistically ‘do-able’. 
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2 BOUNDARIES FOR THINKING 


AND ACTION 
CHRIS BLACKMORE AND RAY ISON 


The focus of this chapter is on using boundaries for thinking and action. It 
is not confined to the process of investigation but stands back from it to 
look at the overall conceptual framework within which investigation takes 
place. It challenges the thinking from which investigative processes start, 
which in turn shapes the design and conduct of an investigation. 


Investigation is an iterative process but once it becomes channelled in a 
particular direction it can be difficult to change course without tools and 
concepts to help. This chapter, in the main, uses the term boundary as a 
‘systems’ concept where a system is a set of components that are intercon- 
nected for a purpose. The boundary is the conceptual line that demarcates 
what is inside the system from what is outside the system, in its environ- 
ment. In other words it separates what is directly relevant for a particular 
purpose from what is not of direct relevance or interest. Should the pur- 
pose change the system boundary needs to be redrawn. Translating this to 
the context of investigation, the suggestion that you use boundaries for 
thinking and action is intended to challenge you always to be clear about 
the purpose of your activity. If your purpose changes you need to recon- 
sider where the boundary needs to be redrawn and what and who should 
and should not be included in your investigation. 


We have interpreted ‘investigation’ quite widely in this chapter as we con- 
sider much thinking and action for policy and development management 
to be part of a ‘finding out’ or a ‘learning’ process. Your own interpretation 
of investigation may be narrower than ours. Don’t use the chapter 
uncritically but do at least try to use it to widen your thinking so that you 
recognize what is relevant in an investigative situation. 


2.1 WHY BOUNDARIES? 


Drawing boundaries is relevant to the development manager or policy in- 
vestigator in many different contexts. They help to separate, simplify and 
focus on what is important in a particular situation and what is less impor- 
tant and can be ignored. For example: 


1 identifying the geographical location of boundaries within which new 
development activity might take place under a policy, project or pro- 
gramme; 

2 identifying people who are involved in an activity who might need to 
participate in investigation and action; 

3 separating your own professional responsibilities from those of a col- 
league — identifying the boundaries of your role; 
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4 working out the expected effects of interventions; and 
5 defining where responsibility and accountability might reside. 


In some situations it might be difficult to place boundaries (not all situ- 
ations are easily bounded or they may be fiercely contested) but thinking 
through where the limits lie can help in planning, challenging, negotiating 
and evaluating many different activities. 


Whilst some boundaries are physical and can be easily seen, others are 
more abstract. They are constructs used to develop and communicate 
understanding and to define and negotiate limits. In this chapter we will 
define boundaries in abstract terms and in relation to the concept of a ‘sys- 
tem’. Thus a boundary is determined by stakeholders (which may or may 
not include you as development manager or investigator) and relevant 
actors when they differentiate a ‘system’ from its ‘environment’ or a ‘sub- 
system’ from the ‘system’. Depending on their perspectives and purposes, 
different people in the same situation will recognize different systems, and 
constituent sub-systems. That is, they will put the boundary in different 
places. 


We shall differentiate between ‘open’ and ‘closed’ systems, which may be 
distinguished by the nature of their conceptual boundaries, Thus a project 
for which there are fixed, perhaps highly specified, outcomes from which 
little or no deviation is anticipated or tolerated, can be seen as a closed 
system with a fixed boundary. In such systems we can say that the bound- 
ary is closed to new inputs to the system, such as the experiences and learn- 
ing of the actors in the process of implementation, or data about changed 
or changing factors in the environment of the system. Projects developed 
within a blueprint approach to development, as described by Korten (1980, 
1990), typify closed systems. 


In contrast open systems have flexible and ‘permeable’ boundaries which 
allow the system to adapt — to respond to inputs and to generate new forms 
of output. This process of change occurs through iteration and learning. 
We would argue that one of the needs in development management is to 
move away from the notion of projects as closed systems towards more 
open systems. Likewise policies which are prescriptive, and are unable to 
be adapted to diverse, local contexts can also be conceptualized as having 
closed boundaries. However, they can be viewed as either open or closed 
systems depending on how they are formulated, articulated and imple- 
mented. This is not an either/or situation but one in which ‘open systems’ 
are the context in which certain ‘closed systems’ might be found. This is 
why development managers need to be able to negotiate, help resolve con- 
flict and evaluate processes and outcomes in order to manage boundary 
formulation and reformulation as an open process. 


An example of this might be a particular investigation as part of the pro- 
cess of policy formulation. An investigation into say, infant mortality in a 


42 


2 BOUNDARIES FOR THINKING AND ACTION 


particular region, as a precursor to the development of post-natal health 
care services policies, may have a fixed time frame and be recognizably 
discrete or bounded. In systems terms we would say that it might be help- 
ful to consider this investigation as a sub-system of a ‘system to develop 
regional post-natal health care services’. Conceptualizing it in this way 
means that the investigation can be seen as part of a greater whole. In turn 
this may help shape the design and conduct of the investigation because 
thinking of it in these terms creates a constant reminder that the investi- 
gation is related to this greater whole — the policy. Of course the policy 
would be unsustainable if it did not contain within it provision for the 
regular collection, analysis and interpretation of infant mortality data. In 
this way it is possible to see what is described as a recursive relationship 
between an initial investigation, policy and the ongoing implementation 
of policy, or one in which feedback from data collection and analysis is 
constantly informing action. 


2.2 USING BOUNDARIES IN DEVELOPMENT MANAGEMENT 


The concept of a ‘boundary’ is fundamental to the study of systems which 
employs systemic or holistic thinking to guide action in complex situations 
(Checkland, 1981). A point which is frequently made in the teaching of 
systems is that our actions in a given context are very much shaped by our 
ways of thinking. Hence it makes sense to think about how we think — what 
particular theories, metaphors, experiences shape how we react, plan, man- 
age etc.? 


Thus, one way of using the concept of ‘boundaries’ is as a tool to guide 
your thinking (examples are given below). A development manager might 
be more interested in linking theory to action for some purpose in a given 
context rather than thinking about theory itself. But this can mask a whole 
set of complex questions: What theories? Who are or should be the actors? 
Who specifies the purpose? What is the context and how is it understood 
and by whom? These are all boundary setting questions that need to be 
considered at the beginning of any investigation and are developed in Box 
2.1. The answers to these questions result in sets of premises for the design 
of a system of interest. 


The checklist of questions in Box 2.1 can be divided into four groups of 
three (Ulrich, 1987). They are: 


* the sources of motivation of those involved — what is the value basis of 
the design? (questions | to 3) 


* the sources of control — who has power or authority and on what basis? 
(questions 4 to 6) 


* what are the sources of expertise or know-how and is it adequate? (ques- 
tions 7 to 9) 


© what are the sources of legitimation and its basis? (questions 10 to 12) 
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BOX 2.1 A CHECKLIST OF BOUNDARY SETTING QUESTIONS 


1 Who ought to be the client (beneficiary) of the system being de- 
signed or improved? 

2 What ought to be the purpose of the system; i.e. what goal states 
ought the system be able to achieve so as to serve the client? 

3 What ought to be the system’s measure of success (or improve- 
ment)? 

4 Who ought to be the decision taker, that is, have the power to 
change the system’s measure of improvement? 

5 What components (resources and constraints) of the system ought 
to be controlled by its decision taker? 

6 What resources and conditions ought to be part of the system’s 
environment, i.e. should not be controlled by the system’s deci- 
sion taker? 

7 Who ought to be involved as designer of the system? 

8 What kind of expertise ought to flow into the design of the sys- 
tem; i.e. who ought to be considered an expert and what should 
be their role? 

9 Who ought to be the guarantor of the system; i.e. where ought 
the designer seek the guarantee that their design will be imple- 
mented and will prove successful, judged by the system’s meas- 
ure of success (or improvement)? 

10 Who ought to belong to the witnesses representing the concerns 
of the citizens that will or might be affected by the design of the 
system? That is to say, who among the affected ought to get in- 
volved? 

11 To what degree and in what way ought the affected be given the 
chance of emancipation from the premises and promises of the 
involved? 

12 Upon what world-views of either the involved or the affected ought 
the system’s design be based? 

(From Ulrich, 1987) 


Of course there are no simple answers to these questions and they are often 
contested and the basis of conflict. Under such circumstances it is neces- 
sary to have some appreciation of power relations (see Chapter 7) and how 
they might be understood. 


For example, John Heron (1989) identifies three levels of power to be con- 
sciously recognized in the process of project or activity design: 


44 


2 BOUNDARIES FOR THINKING AND ACTION 


1 hierarchical, with ‘power over’ leading to ‘deciding for’; 
2 co-operative, or ‘power with’ leading to ‘deciding with’; and 
3 autonomous, or ‘power to’ leading to ‘delegating deciding to’. 


Choice of any one of these frameworks will lead to very different out- 
comes in a given context because the systems that result will be very dif- 
ferent. There is no right answer as to which conception of power might be 
used in a given context but these distinctions do exemplify: 


(a) how the boundaries around a given system of interest might vary de- 
pending on the perspectives which different participants bring; 


(b) how this might be consciously incorporated into process design; and 


(c) how open to conflict and negotiation the whole process might be un- 
derstood to be. 


2.2.1 ACKNOWLEDGING INDIVIDUAL PERCEPTIONS AND PERSPECTIVES 


The point has been made that individuals confronted by the same situation, 
are likely to recognize different ‘systems’ because they have different per- 
spectives. It is worth considering this notion of perspectives further. When 
we stop to think about it we know that each of us has a unique experiential 
history — even within families, groups or cultures no human beings share 
exactly the same experiences. From this unique cognitive history it fol- 
lows that all we have at our disposal is the ability to communicate about 
our experiences. We never have exactly the same experience as another. 
Thus we each bring to any situation, and into any conversation, sets of 
unique perspectives. There is a tendency to forget this, although when re- 
minded it seems like common sense. 


Unfortunately many factors act against us remembering that human com- 
munication lies in a complex process. One of the most powerful constraints 
is the common metaphors associated with human communication — e.g. 
‘information transfer’, ‘information sharing’, ‘getting the message across’. 
Hidden in these metaphors is the notion that human communication is just 
like two computers linked to each other, i.e. that the message sent will be 
exactly the same as the message received. Clearly however, human com- 
munication is not the same as two computers ‘talking’. We can never be 
sure that the meaning attributed to what we said is the same as we in- 
tended. In fact the meaning attributed will be an interpretation of the mes- 
sage from the unique perspective of the other person. 


Recognizing that human communication is often misunderstood and that 
some pervasive metaphors seem to conspire to perpetuate this misunder- 
standing enables us to understand the importance of designing and devel- 
oping effective communication processes. As discussed further by Chataway 
and Joffe in Chapter 10, we cannot think in terms of investigation as lead- 
ing to results which are then simply transmitted to policy-makers who act 
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on them. It is better to think of an investigation as including writing up and 
feeding back results to particular audiences, and this whole process as a 
negotiation — including the changed meanings constructed by the mem- 
bers of the audience as they interpret the investigation and its results from 
their own unique perspectives. 


2.2.2 MOVING BOUNDARIES TO GET OUT OF TRAPS IN THINKING AND 
BEHAVIOUR 

Many boundaries are those of our own making and they affect the way we 
think and behave. For example, at some time in your life you have prob- 
ably experienced a situation where you constructed a boundary between 
what you could and could not do, but later managed to overcome fears, 
develop skills or see different opportunities that enabled you to change the 
boundary. There might be no physical reason why you can’t speak up in a 
public meeting, jump across a ditch or travel to another country, but if you 
tell yourself you can’t do it, you often won’t be able to. If however you can 
begin to think of a task differently you might find ways you can do it. This 
process is one of moving the boundary between what you perceive you 
can and cannot do. 


In the context of development management, perceptions of resource limi- 
tations provide an example. Resources are elements which are available 
within the boundary of a system. Before working out where the bounda- 
ries lie, it is important first to identify what system or sub-system you are 
considering. Take the case of a team of three busy people with a small 
fixed budget, tasked with running a major series of training events as well 
as maintaining an existing activity. If the boundary of project resources 
were drawn around these elements the task would seem unachievable. How- 
ever, if other resources can be identified — involving other people, sharing 
the task with another team, obtaining additional funds from those with a 
complementary agenda — then it might indeed be achievable. In this case it 
is only the system for organizing training events we need to consider, not 
the system for running the project as a whole. Working out where the par- 
ticular activity boundary might lie (in this case training) rather than the 
project boundary enables us to consider the situation quite differently. 


One common trap in using systems thinking is the belief held by many 
scientists or development managers that it is possible to stand outside any 
‘system’ in a so-called objective position. This is often manifest in behav- 
iour in which one person or group claims to have access to the ‘truth’ 
rather than recognizing that in a given situation there may be many ‘truths’. 
The relationship between individual and collective experiences with re- 
spect to investigation for action is another area in which it may be useful to 
think in terms of how people construct and draw boundaries around knowl- 
edge. 


For example, knowledge can be seen as a commodity or a body of 
understandings (based on data, analysis and interpretation) which can be 
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passed on in verbal or written form to others. Knowledge can also be seen 
as a process of ‘knowing’ which is very personal and involves experiential 
learning. Moreover, different people will place different emphasis on sci- 
entific and traditional perspectives. Redclift (1992) points out that when 
we observe local resource management strategies we must acknowledge 
that we are dealing with multiple epistemologies (knowledge systems) 
possessed by different groups of people. There are many different ways of 
interpreting a statement such as ‘we do not have that knowledge’. It could 
range in meaning from ‘we do not have that data’ to ‘we have not had that 
experience’. Finding out what individuals understand by knowledge — where 
they draw the conceptual boundary — can help to communicate different 
perceptions and build a more coherent understanding. It is in turn a way of 
finding out what action might be taken. 


These different ideas about knowledge and ways of drawing boundaries 
round knowledge suggest that any attempt to ‘find out fast’ should be open 
to the many possible stories or interpretations of a scenario or situation. 
For example, in many parts of Africa and other countries with semi-arid 
rangelands where pastoralism is the most common land use, there has been 
acrimony between scientists and pastoralists for many years focused around 
the so-called ‘land degradation debate’, This is a contest between different 
epistemologies but one in which the players are often unaware of their 
own epistemological positions. In Botswana, Louise Fortmann’s case study 
of rangeland use over a 50-year period shows how official policy consist- 
ently defined the major problem of the pastoral regions as overstocking 
leading to certain ecological disaster (Fortmann, 1989). For the officials 
the problem was clear, as was the technical solution (destocking). Local 
experience on the other hand defined the problem as too little land. The 
local solution was also very different: renting, or using land previously let 
to a European mining company. The local experience was that the local 
range could and did carry an increased cattle population and that besides 
localized problems, the dire official predictions of ecological disaster did 
not happen. While there is general agreement that the quality of the en- 
vironment (as indicated by the quality of the grazing, the number of trees 
and the extent of erosion) is deteriorating, there is clearly no agreement on 
causes or solutions. It is significant that the story has been told consist- 
ently from both perspectives for 50 years: this shows how different and 
how unconnected traditions of knowledge or understanding can be (Russell 
and Ison, 1993; Ison and Russell, 1998). 


Stephany Kersten, in an Australian study (Kersten, 1995), found a similar 
contest between researchers and pastoralists over the nature and causes of 
the so-called ‘rangeland degradation problem’. In her research she used 
particular techniques to design participatory workshops with the aim of 
enabling researchers, pastoralists and advisors to break out of this histori- 
cal trap. One approach she adopted was to invite the participants to draw 
pictures, using mind-maps or spray diagrams, of who they were as people. 
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One of the diagrams drawn by the participating landcare co-ordinator (an 
employee of a local community Landcare Group, funded by the govern- 
ment) is shown in Figure 2.1. 
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FIGURE 2.1 Spray diagram drawn by the Landcare Facilitator; CalM is Conserva- 
tion and Land Management (source: Kersten, 1995) 


By doing this Kersten was inviting the participants to move their concep- 
tual boundaries; from belonging to, or representing groups of researchers, 
pastoralists or advisers, to a large single group of individuals coming to- 
gether as individual people with a common concern about the future of the 
rangelands. Her research showed that this was one necessary precursor to 
the generation of dialogue (as opposed to debate). 


Some differences between debate (the original meaning of which can be 
interpreted as ‘to put down’) and dialogue (“has meaning running through’) 
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are given in Table 2.1. Dialogue in this sense is an ideal which cannot 
always be realized, and it is often necessary to think in terms of negotiation 
between interests as another alternative to ‘debate’ (see also Section 2.2.4 
below). Factors that enhanced or restricted dialogue, arising from meet- 
ings between pastoralists, researchers and advisors, are summarized from 
Kersten’s research in Table 2.2. The research by Kersten is a good example 
of the design of a communication process which recognizes the multiple 
perspectives that we bring to our conversations. 


TABLE 2.1 SOME OF TH 


ETWEEN ‘DEBATE 


Debate 


Participants tend to be leaders of 
groups with a specific opinion, 

to speak as representatives of a 
group, or be committed to a 
particular view 

The atmosphere is threatening, 
attacks and interruptions are 
expected 

Differences within the group are 
set aside or denied 

Participants listen in order to refute 
other ideas, questions are often 
rhetorical challenges or disguised 
statements 


Statements are predictable and 
offer little new information 


Success requires simple 
impassionate statements 


Participants operate within the 
constraints of the dominant public 
discourse, which defines the 
problem and the options for 
resolution; this assumes that 
fundamental needs and values 

are already clearly understood 


(Kerston, 1995; after Roth et al., 1992. p. 44) 


Dialogue 


Participants are not necessarily 
leaders, they speak as individuals 
about their own unique experience, 
expressing uncertainty, as well as 
deeply held beliefs 


The atmosphere is one of safety, 
ground rules exist and promote 
respectful exchange 


Differences between individual 
participants are revealed 


Participants listen to understand 
and gain insight into the beliefs and 
concerns of others; questions are 
asked from a position of curiosity 


New information surfaces 


Success requires exploration of the 
complexities of the issue being 
discussed 


Participants are encouraged to 
question the dominant public 
discourse, to express fundamental 
needs that may or may not be 
reflected in the discourse, and to 
explore various options for problem 
definition and resolution 
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Enhancing dialogue Restricting dialogue 

Participants come to a meeting as Participants come as representatives 

individuals of a group 

Participants articulate their personal _ Participants are at the meeting as 

understanding at the meeting groups and participants act as part of 
their own group 

Time has been spent on building Little time has been spent on 

relationships before the meeting building relationships 

and during the meeting 

Participants are prepared to relax Participants have fixed general or 

preconceived ideas about other stereotyped ideas about other 

participants at the meeting participants 

Participants do not know each other _ Participants know each other before- 

beforehand hand and are not prepared to relax 
preconceived ideas about each other 

Listening actively to other Participants listen to re-establish 


participants with an open mind that _ preconceived ideas 
is not blocked by preconceived ideas 
Participants are open to ideas and Participants are defending or 


asking for suggestions of other attacking statements made 
participants 

Participants respect other meanings _ Participants do not respect meanings 
and understandings. Multiple and understandings other than their 
realities are acknowledged own. They believe in one reality 


Participants feel they can benefit People have the feeling they are 
from a good discussion with people —_ ‘being participated’ 

who see the same issue from 

different perspectives 


(Kersten, 1995) 


Another factor which affects the way people view the world and set bounda- 
ries is their profession. One of the more common traps is narrow disciplin- 
ary thinking. Many professionals see the world in the light of their disci- 
plinary perspective and all too often have a tendency only to recognize 
problems and opportunities accordingly. For example Figure 2.2 shows a 
grassland system and how different experts view the system, depending 
on their area of expertise. Such thinking has caused many difficulties in 
project formulation and is a strong argument for the use of interdisciplin- 
ary perspectives and systemic (or holistic) as opposed to systematic (step- 
by-step) methodologies in the project formulation process. Note that in 
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interdisciplinary team 


FIGURE 2.2 Perspectives of a farming system taken by disciplinary experts differ 
from that taken by an effective interdisciplinary team (Adapted from U. Schevermeier, 
unpublished) 


Figure 2.2 the perspective of the interdisciplinary team reflects far more 
fully the grassland system as a whole. 


Another example is that in completing the logical framework plans used 
by certain development agencies, many disciplinary ‘natural’ scientists 
consistently name organizational, institutional and other socio-economic 
factors as likely or possible constraints to ‘natural science’ research suc- 
cess. In other words many of the complex people- and policy-related 
issues are seen as outside the system boundary by those who proposed the 
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projects. The use of multi-disciplinary teams in rapid rural appraisal (RRA) 
and participatory rural appraisal (PRA) constitute attempts to break out of 
such disciplinary traps, though these methodologies have their own prob- 
lems. (See Chapters 6 and 7.) 


Historically much agricultural and rural development has been managed 
by scientists and technologists, including agricultural economists who have 
focused on the concept of the ‘farm’. This has been exemplified in ‘farm 
management’ and ‘farming systems research’ (FSR) where the tendency 
has been to define the boundary in terms of a ‘farming system’ whilst ne- 
glecting the wider rural context. For example, seeing the desired system as 
a ‘tural livelihood system’ would also allow artisanal, craft and wage la- 
bour to be considered within the system boundaries. In recent years FSR’s 
remit has expanded tremendously to include this wider perspective. Figure 
2.3 illustrates how farming systems are now seen as sub-systems that con- 
tribute in one way to ‘rural livelihood’ or in another to ‘food’ systems 
(Bebbington et al., 1993; Bebbington, 1994). 
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storage network 


FIGURE 2.3 A systems diagram of a farming system as a sub-system both of a rural 
livelihood system and of a food system 


Development managers can make important contributions to the investi- 
gation and design of interventions which enable clients or stakeholders to 
break out of traps or self-imposed boundaries to their thinking that they 
themselves may not appreciate..We believe that this is most effective when 
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the participants are enabled to take responsibility for their own learning 
and insights (McClintock and Ison, 1994). Such enabling processes re- 
quire a high level of self-awareness among development managers them- 
selves (see Chapter 7). 


2.2.3 BOUNDARIES TO SIMPLIFY AND COMMUNICATE 


Many situations under investigation by development managers are com- 
plex, messy and apparently unbounded. Drawing boundaries and diagram- 
ming can help to simplify a situation and act as a powerful means to com- 
municate different perspectives among participants as we saw in the study 
by Kersten. There are many forms of diagramming which could be part of 
any development managers ‘tool-kit’. One of the simplest is the systems 
diagram (or Venn or ‘chapati’ diagram), of which Figure 2.3 is an example, 
which shows what is considered to be inside and outside particular system 
boundaries and how systems and sub-systems overlap. Some other useful 
diagramming techniques are given in Figure 2.4; the figure is itself an ex- 
ample of a tree diagram. These and other forms of diagramming approaches 
are used extensively within RRA and PRA, while Chapter 7 includes sev- 
eral examples of ‘mental maps’. 


diagrams 
fon-pictorial (partly) pictorial 
representations 
static diagrams 
3 temporally ordered rich various 
(or pataly ordered) pictures —_ technical 
classification structure and ad illustrations 
diagrams + relationship 
\ diagrams 
‘ flow causal graphs 
matrices spray diagrams diagrams (with time 
a, diagrams dimension) 
trees ; 
a influence 
fe ‘, diagrams 
pees ‘ sign multiple- 
bal aia ‘\ information/ pate ibe 
Gene me materials activity diagrams 
5 ost etigiflOW sequence 
diagrams 
control cpa decision 
del Lageleeatt networks sequence 
papa dynamics diagrams 
diagrams — diagrams 
algorithms 


FIGURE 2.4 Different diagramming techniques available to the development 
manager for use during investigations (Open University, 1996) 
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Diagrams can be a very powerful means of cross-cultural communication 

although their effectiveness in communication differs depending on whether 

(a) the diagramming is done as part of a process in which there is joint learn- 
ing and appreciation of why things were done in a particular way; or 

(b) the completed diagram is simply presented to an audience, which is a 
qualitatively different experience. 


An important point to remember in developing systems diagrams is that 
they should not be restricted to what already exists. In fact they lose much 
of their utility if they do not picture ‘what might be’ as well as ‘what is’ and 
thus may also include components and linkages which do not actually exist 
in the situation under consideration. When used in the early stages of any 
investigation, systems diagrams, as well as other diagramming and crea- 
tivity techniques, are an important means of finding out fast. 


Development managers undertaking an investigation may produce their 
own completed systems diagram as a way of simplifying a complex situa- 
tion and providing a basis for discussion. (One example of a diagram used 
as a stimulus for discussion, and with reactions providing new information 
for the investigator, is Figure 8.3 in Chapter 8.) 


Alternatively, diagramming may be done collaboratively, as a tool for ne- 
gotiation. Development managers have a responsibility to try to be trans- 
parent in the boundary judgements they make and to trace their possible 
consequences. Ulrich (1987) argues that planners and managers cannot take 
upon themselves the ‘political’ act of approving the consequences of their 
boundary making. We would like to agree with Ulrich and say that only the 
affected can do this. However, this would seem to negate the realities of 
power in many situations. 


The development of conceptual models through the process of diagram- 
ming is also a valid means to start the policy formulation process. Some 
approaches to systems modelling stress the importance of using verbs — 
‘doing’ words — as the modelling language (e.g. gather data, analyse data, 
identify stakeholders, see Checkland and Scholes, 1990). Particular func- 
tions are then incorporated within the system through the choice of appro- 
priate boundaries. For example, Figure 2.5 was used as part of the process 
of formulating a livestock sector development programme in Nepal. The 
figure also makes the system a lot easier to conceptualize than a descrip- 
tion of the same model in words: 


a system to influence the living standards of people in Nepal’s 

rural communities by concurrently developing the productivity 

of the livestock sector in an environmentally sustainable and 
equitable manner, while enhancing the capacity of the people to 
manage an on-going process of development. The system is 
dependent on long-term policy support from government and donor 
agencies and requires the development of institutional arrangements 
and capacities to facilitate people-centred development. 
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FIGURE 2.5 A form of conceptual modelling, using verbs as the modelling 
language, which is based on soft-systems methodology (Macadam et al., 1995) 


2.2.4 BOUNDARY SETTING AS A PROCESS FOR NEGOTIATION 


Getting people to explain — either verbally or visually — what they consider 
to be inside or outside a boundary is a starting point either for ‘dialogue’ or 
for ‘negotiation by interests’ as discussed in Section 2.2.2. How this is 
done needs careful thought. Slim and Thompson (1993) stress the impor- 
tance of listening to the voice and experience of people particularly in 
cultures with a strong tradition of oral testimony. Table 2.3 shows how 
Chambers (1992) highlights some differences between verbal and visual 
modes in the context of participatory rural appraisal. The right-hand col- 
umn shows the potential of participative group-based techniques using 
visual representations particularly when, in Chambers’ terms, the outside 
investigator ‘hands over the stick’, but this potential is not always realized 
(see Chapter 7). Chambers points out that the verbal will always remain 
important and suggests that a combination of the verbal and the visual can 
express more than either on their own. 
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TABLE 2.3 SOME CONTRASTS BETWEEN VERBAL AND VISUAL 


MODES OF WORKING COLLABORATIVELY 


Verbal Visual 
(interview, (diagram) 
conversation) 
Outsider’s roles Investigator Initiator and 
catalyst 
Outsider’s mode Probing Facilitating 
Outsider’s interventions Continuous and Initial and then 
maintained reduced 
Insider’s roles Respondent Presenter and 
analyst 
Insider’s mode Reactive Creative 
Insider’s awareness of outsider High Low 
Eye contact High Low 
The medium and material are 
those of: Outsider Insider 
The poorer, weaker, and women 
can be: Marginalized Empowered 
Detail influenced by Etic categories Emic 
(imposed from perceptions 
outside) (constructed 
from inside) 
Information flow Sequential Cumulative 
Accessibility of information to Low and transient — High and semi- 
others permanent 
Initiative for checking lies with Outsider Insider 
Utility for spatial, temporal and Low High 
causal information, relations, 
analysis, planning and monitoring 
Ownership of information Appropriated by Owned and 
outsider shared by 
insider 
(After Chambers, 1992) 


People who are in actual or potential conflict can often reach a better un- 
derstanding of each other’s point of view by working together and modify- 
ing the boundaries of their views. In some cases people may find they have 
closer or more complementary views than they think. An example where 
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this has been done is in the UK where the ‘Planning for Real’ approach has 
been used in housing areas in Glasgow, Leicester, Nottingham, Tyneside 
and Sheffield (Gibson, 1994). Three-dimensional models with movable 
parts were used as part of the methodology so that participants could ex- 
periment with different scenarios and design their future amenity areas 
together. The process started with residents’ suggestions. Officials and poli- 
ticians were then drawn in to add their advice and experience, for example 
of legal and technical constraints, costs of proposed developments and 
sources of money and materials. Several stages of working out priorities 
and resources followed the modelling. No limits were put on the number 
of initial suggestions but part of the process involved removing duplica- 
tions and having second thoughts about proposals that on reflection didn’t 
look so good after all. In some cases this approach provided a way for 
residents to work with officials and politicians with whom they had previ- 
ously been in conflict. Overall, residents used the technique to explore 
their own ideas and to negotiate with each other and the officials to find the 
limits of what is acceptable and desirable to all those involved. 


2.2.5 THE NEED FOR ITERATION IN BOUNDARY SETTING 

Views, understandings, values and beliefs are not static. The experience of 

engaging in discussion and dialogue, listening to others and deepening 

understanding and negotiation is dynamic and the boundaries constructed 

will change. For development managers this raises two important issues: 

(a) the need to consider boundary setting as a continuous or iterative pro- 
cess; and 

(b) who are relevant stakeholders in the construction of any system and 
how might they be involved in the boundary setting process? 


There are various tried and tested ways of working out what system you 
need to consider and where its boundaries lie so that you can think about it 
in different ways, build understanding and work out appropriate forms of 
investigation and action. Facilitating a meeting where different perspec- 
tives and ideas can be heard, brought together and built on, is one such 
way. Such an approach places value on diversity of perspectives and data, 
in contrast to many standardized modes of investigation which seek to hom- 
ogenize and normalize. Using more formal methodologies such as soft sys- 
tems methodology (Checkland and Scholes, 1990) and participatory rural 
appraisal (Chapters 6 and 7) are other possibilities. 


It is not possible here to expand on soft systems methodology (SSM) in 
detail. It is among the many formal methodologies which are potentially 
useful to the development manager or policy investigator in finding out 
fast. It was developed (Checkland, 1981) as an approach to resolving com- 
plex, human-centred problems where there was agreement about neither 
the problem nor the possible solutions. Elements of SSM have been used 
creatively by ActionAid in Somaliland to negotiate and renegotiate the 
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EXAMPLE 2.1 USE OF SOFT-SYSTEMS METHODOLOGY 
BY ACTIONAID SOMALILAND 


ActionAid Somaliland (AAS) used a modified soft-systems method- 
ology (SSM) to generate their strategic objectives. This is a descrip- 
tion of one strategic objective, written as a system of interest or root 
definition, and which contains reference to the elements of the mnem- 
onic CATWOE. The activities which AAS will carry out to imple- 
ment this system are also listed. 


Strategic objective 

An AAS-owned system, managed by traders and herders, to make at 

least five essential veterinary drugs available to the traders, PVAs 

(primary veterinary assistants) and herders, so that livestock remains 

healthy and food security of the region is maintained. 

Elements 

C client: traders, PVAs, herders. 

A actors: AAS, ODA, herders’ fora, traders and NGO committee. 

T transformation: non-availability of the essential veterinary drugs 
to availability of five essential drugs. 

W world-view: food security in the area depends on livestock, so that 
it is necessary to keep the livestock healthy. 

O owner: AAS. 

E environment: lack of enough experience of veterinary-drugs trad- 
ing business by the Sanaag community, poor communication fa- 
cilities with the outside world and security. 


Activities 
1 AAS provides communication facilities to allow the traders to be 
linked to outside drug companies. 


Nn 


AAS shares experience of the progress and constraints of the pro- 

gramme with the other international organizations involved in this 

sector. 

3 AAS stores a stock of essential drugs for emergency purposes. 

4 AAS provides information on the veterinary drugs available in the 
region to the herders and PVAs. 

5 AAS provides information on the diseases existing in any given 

period of the year to the traders. 
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boundaries to systems of concern to them. Example 2.1 shows the use of a 
statement about a ‘system of interest’ (called a root definition) which is 
generated in a group process going through several iterations. During the 
process the mnemonic CATWOE has been used as a tool to check that the 
description of concern has all the elements necessary for the subsequent 
conceptual modelling of this system of interest. Research has shown that 
the best statements refer to 


* Clients (C); 
* relevant Actors (A); 


* the Transformation (T) carried out by the system (in this case non- 
availability is transformed into availability of drugs by the system); 


* the World-view or views (W) held by those developing the system; 
* Owners (O) of the system; and 
* the Environment (E) in which the system will exist. 


The process of refining the root definition and checking it using CATWOE 
statements is an iterative process. In this example it has enabled ActionAid 
to develop boundaries around five specific activities that need to be car- 
ried out to meet one of its strategic objectives. ActionAid has also used 
SSM to evaluate its activities from three perspectives: its own; the funders 
Overseas Development Administration (ODA; now Department for Inter- 
national Development, DfID); and local pastoralists. 


Such participatory multi-perspective evaluations are likely to increase in 
future. They are another example of an open system — the process reveals 
the many trade-offs and contrasting judgements of what is, or is not, a 
success. It also sees evaluation as starting at the beginning of any policy 
or project development activity and continuing through the life of the 
activity. 

One of the important features of SSM is its focus on iteration — of ‘learn- 
ing your way’ to new appreciations of complex situations. 


2.3. WHO PARTICIPATES IN BOUNDARY SETTING? 


As we have already seen, where boundaries are drawn depends a lot on 
who is involved in drawing them. Concepts are frequently understood dif- 
ferently by different people; for example, the notion of family or house- 
hold. As Feldstein and Jiggins (1994) point out, one of the most important 
starting points is deciding who to ask to draw a particular boundary. Thus, 
for example, explicitly asking women at the start makes it easier to incor- 
porate gender concerns into any boundary construction. Asking men and 
women to draw boundaries around the same concept may also reveal dif- 
ferent perceptions (see Chapter 7). This relates back to stakeholder analy- 
sis and the processes by which stakeholders are identified and perhaps 
invited to participate (see Chapter 8). 
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Boundaries can help us to work out who is or should be involved in a given 
process, such as decision-making. An example is provided by Uphoff (1986, 
1992) when considering the types of local institutions involved in develop- 
ment activity (see Figure 2.6; each level represents a boundary identified 
as being of relevance). 


international level 


eet aie: 


national level 


regional/provincial 
level 


aw aa 


district level 


atin el eee 


sub-district level 


a set of communities having social and economic 


locality level relations; this is the same as the sub-district level 


= where a market town is the sub-district centre 


: a relatively self-contained socio-economic residential 
community level unit 


local institutions 


a selfidentified set of persons with some common 
group level interests: may be persons in a small residential 

area like a neighbourhood, or an occupational, 
age, gender, ethnic or other grouping 


household level 
Ss 


individual level 
ao a See 


FIGURE 2.6 Ten levels of decision-making and activity (After Uphoff, 1986, p.11) 


Uphoff uses boundaries in two ways to analyse and explain what he means 
by local institutions. Firstly he distinguishes between /Jevels of decision 
making and activity — ranging from individual to international. He con- 
structs boundaries around different groups of decision-makers and activi- 
ties in order to define these levels and by naming and describing the levels 
he is able to communicate his understanding to others. Secondly, he chal- 
lenges the way many people use the word Jocal by constructing a further 
boundary around three of the levels — the locality, community and group 
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levels. The distinction is important because, for example, the involvement 
of local level institutions is recognized (by Uphoff and others) as essential 
for mobilizing resources and helping to resolve resource management con- 
flicts in an adaptive way. Picking out just one of the three local levels for 
decision-making may well mean that opportunities are missed. 


Such an approach to boundary setting can be helpful for investigation. It is 
often extremely important to know what the potential levels of any inves- 
tigation are and how boundaries might be drawn around them to: 


* challenge accepted views where appropriate; 

* act as a means of communication to those who are participating in the 
investigation; and 

* decide at which level or levels it is appropriate to carry the investigation 
out. 


2.3.1 KEEPING BOUNDARIES OPEN 


The rapidly changing nature of the challenges of development manage- 
ment means that there is a conscious need to keep boundaries open. Many 
factors relating to projects and policies will change over time. Different 
people, who offer new perspectives, will be involved. And there will be a 
need to respond — both proactively and reactively — to events that occur 
and resources that become available (or unavailable). There is a need for 
iteration — for the ‘system of interest’ to be constantly constructed and 
reconstructed. Otherwise, there is a tendency to work with closed systems 
or blueprints without reference to the wider dynamics (or open systems) of 
which they are a part. In development, this can lead development manag- 
ers to think of targets (human and physical) rather than processes. 


NGO involvement in the United Nations Commission for Sustainable De- 
velopment (CSD) process of follow up from the Rio Earth Summit pro- 
vides an example of action based around policy. The CSD was set up by 
the UN Economic and Social Council (EcoSoc) to monitor progress on the 
resourcing and implementation of the Summit’s major plan of action, 
Agenda 21, and to make recommendations on it through EcoSoc to the UN 
General Assembly. Although the system might appear closed in that there 
are many rules and regulations (originally set down after the Rio confer- 
ence) about who can participate, both the governments and the NGOs as- 
sociated with the CSD have set out to facilitate involvement of stakeholders 
and enable it to be an open system. The CSD broke down Agenda 21 so 
that it could be reviewed over the three-year period 1994-1996 and NGO 
involvement has been continually negotiated, with the result that since the 
Earth Summit the role and access of NGOs and ‘major groups’ (see below) 
in the CSD process has increased greatly. The facilitation role played by 
individuals within the UN, governments and some NGOs to ensure that 
individuals who can bring relevant new perspectives are able to participate 
effectively might be thought of as ‘demystifying’ the process or interpret- 
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ing the rules. It can also however be thought of in terms of recognition of 
the need for iteration, acknowledgement that the system of interest is chang- 
ing as different areas come up for review, and thus, questioning and repo- 
sitioning the boundaries. 


Some individuals within the international NGO community have been par- 
ticularly active in this process and this example highlights the way in which 
skilful facilitation can change a potentially closed system into a more open 
one. Another point illustrated by this example is that it is not enough to rely 
on self-selection for participation in formulating policies and projects. It is 
often necessary for an intervention to be made to encourage and facilitate 
participation of groups that frequently do not get heard in a process. Some 
of these are included in the ‘major groups’ involved in the CSD: women, 
children and youth, indigenous people, NGOs, local authorities, workers 
and trade unions, business and industry, the scientific and technological 
community and farmers. It is interesting to trace the reasons why these 
groups were selected above others. It was due at least in part to them or- 
ganizing themselves, or being helped by others, so that they had a voice in 
the process. Boundaries for participation can also be changed by ensuring 
that different channels of communication are taken into account. The im- 
portance of oral testimony and listening must not be forgotten in policy 
and project formulation processes such as the CSD, especially where there 
is heavy reliance on written material. However it is unlikely to be accounted 
for unless facilitated for example as in Section 2.2.4. 


Stakeholder analysis (see Chapter 8) is one technique that can be used to 
work out who should and who does participate in formulating policies and 
projects — it is a boundary setting process. To date it has not been well 
developed for policy formulation but has had increasing use in project for- 
mulation (e.g. Grimble et al., 1995; Montgomery, 1995a and b; Grimble 
and Wellard, 1996). A challenge for the future is how to create the political 
and institutional space that might enable stakeholder analysis to be used 
more in the policy formulation process. Such a move would challenge ex- 
isting conceptions of time (e.g. the common three-year project), value (e.g. 
a focus on outputs as opposed to greater focus on the design stages) and of 
course vested interests, including the kinds of relationships between do- 
nors and recipients. 


Keeping boundaries open is also important for conceptualizing and 
reconceptualizing investigative processes. One area in which this has been 
attempted is that of investigation or data collection for impact assessment. 
‘Impact assessment’ is a term applied to a whole range of techniques that 
have evolved from environmental impact assessment (EIA) which was first 
introduced in the USA in 1969. This evolution was itself due to changes in 
perceived boundaries of development activities. Different dimensions have 
been incorporated to meet a growing range of needs around predicting the 
effects of development activities. 
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A directory compiled by Roe, Dalal Clayton and Hughes (1995) for the 
International Institute for Environment and Development’s (ITED) Inter- 
national Environmental and Natural Resources Assessment Information 
Service (INTERAISE) project gives details of: 

* environmental impact assessment; 

* cumulative effects assessment; 

* (environmental) health impact assessment (E)HIA; 

* risk assessment; 

* social impact assessment; 

* strategic environmental assessment; 

* technology assessment. 


These techniques are useful for a range of different purposes but all in- 
volve elements of boundary setting and resetting. In general they all in- 
clude a preliminary stage of scoping — a process of defining the scope of 
the full study in terms of which geographical area, which types of impact 
and which affected groups should be included in the study, as well as the 
minimum impact to be regarded as significant. This implies deciding what 
is outside the boundary of the study, in other words not worth looking at 
further. Thus the process of scoping in EIA should involve exploring and 
identifying key issues under investigation with interested parties. There 
are different ways in which this process is interpreted, particularly around 
who is involved. One drawback is that it can provide opportunity for vested 
interests to surface (though one could also argue that, if vested interests are 
there, an investigator needs to be aware of them). However, scoping can 
also provide opportunities for information, ideas and proposals to emerge 
and for building an understanding of the impact of given processes based 
on a wide range of experience. Development managers involved in this 
process can play an important role in facilitating local community involve- 
ment and helping to ensure that boundaries are kept open and that there is 
scope for iteration. 


2.3.2 MANAGING PERSONAL AND ETHICAL BOUNDARIES 

It is desirable that development managers and policy investigators are able 
to identify and articulate their own personal and ethical boundaries. Robert 
Chambers (1993) has pointed out how many professionals may be part of 
the problem rather than part of the process of generating improvements. 
Development is replete with examples of those who have come to help but 
who have made things worse. From a systems perspective it is necessary to 
acknowledge that in collaborating with others there is a mutual dance of 
responsibility. The blurring of the relationships and thus boundaries in this 
process has too often led to development practitioners taking responsibil- 
ity for others such as local people, clients, staff etc. The contrasting posi- 
tion is to attempt to design contexts in which each is ‘response-able’ — able 
to respond for themselves (McClintock and Ison 1994). 
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TABLE 2.4 F TYPES OF BOUNDARY WHERE FEELINGS NEED TO BE MANAGED 


Key Boundary Necessary Characteristic feelings 


questions 


tensions 
(positive) 


(negative) 


Who is in 
charge of 
what? 


Who does 
what? 


What is in it 
for us? 


Who is (and 
isn’t) with us? 


trustful 
open 


How to lead 
but remain 
open to 
criticism 


Authority 


How to follow 
but still to 
challenge 
superiors 


Task confident 
competent 


proud 


How to depend 
on others you 
don’t control 


How to 
specialize yet 
understand 
other people’s 
jobs 


How to defend 
one’s interests 
without 
undermining 
the organization 


Political empowered 


treated fairly 


How to 
differentiate 
between win- 
win and win- 
lose situations 


How to feel 
pride without 
devaluing 
others 


Identity proud 
loyal 


tolerant 


How to remain 
loyal without 
undermining 
outsiders 


(After Hirschorn and Gilmore, 1992) 


rigid 
rebellious 
passive 


anxious 
incompetent 
ashamed 


powerless 
exploited 


distrusting 
contemptuous 
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Personal and ethical boundaries have a particularly important role to play 
in investigative processes especially when they are linked to policy devel- 
opment. One aspect is the relationship between the development manager 
and/or investigative team and those communities, institutions and individu- 
als who are helping to provide information or are participating in some 
way in the investigation. The boundaries may be ones of ‘expertise’, edu- 
cation, social position etc., issues discussed earlier in this chapter. Deci- 
sions to blur these boundaries may result in expectations about action or 
outcomes that are impossible to meet, while the ‘reversals’ discussed by 
Chambers and others in using participative approaches (see Chapter 7) can 
also be difficult to achieve. 


Another aspect is the relationship between the development manager or 
investigator and the individuals and institutions involved in policy formu- 
lation. Do boundaries that might exist or be constructed allow develop- 
ment managers or investigators either to challenge or to collaborate with a 
given policy direction? Such challenges or collaboration may be within 
their own institution or take place in a wider institutional arena, and may 
raise complex personal and ethical issues. 


One approach to managing such situations has been suggested by Hirschorn 
and Gilmore (1992). They argue that managers should use their own feel- 
ings as the primary tool for managing boundaries. Table 2.4 outlines some 
of the key questions, boundaries, tensions and feelings that may well apply 
to situations that development managers find themselves in. We have all 
had experiences of working effectively in groups and of the satisfaction 
that comes from this — these are the positive experiences that are possible. 
If a group is not working effectively and negative or ‘bad’ feelings pre- 
dominate then we need the skills to encourage a process whereby these 
feelings can be discussed openly and strategies developed to make changes. 
This means being in touch with our own feelings and being able to deci- 
pher frustrating and difficult personal relationships and diagnose why they 
have gone wrong. 


2.4 CONCLUSION 


Many of the ideas and examples outlined in this chapter could be articu- 
lated in other terms besides using the concepts of ‘system’ and ‘boundary’. 
Such is the nature of constructs. However, they encourage a systemic ap- 
proach to investigation and action which helps make sense of complex and 
messy situations. Vickers (1983) has observed that: ‘if systems thinking is 
to be used appropriately as a tool of understanding then the systems we 
concern ourselves with cannot be arbitrary constructs but must include the 
minimum number of constituents and relationships needed to describe that 
situation’. 

Thus when conceptualizing at the start of a policy investigation it is useful 
to think in terms of different possibilities for placing boundaries as well as 
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in terms of different possible research questions (Chapter 1). Reconcep- 
tualizing can mean a new choice of boundaries and new questions, but it is 
often necessary if an investigation is to be effective (Chapter 3). 


It is not always easy to identify the minimum number of constituents and 
relationships Vickers refers to but the process of considering what systems 
and sub-systems are relevant to a situation can help to draw out those as- 
pects that have been omitted or should be added. ‘Systems thinking’ in- 
volves considering a number of partial and simplified views of a situation 
to better understand the whole situation. Focusing on boundaries is a par- 
ticularly useful way forward in many of the examples given because it 
enables those involved to both think and act differently and to recognize 
that investigation and policy formulation are iterative processes. It particu- 
larly helps us to avoid the either/or trap associated with the distinctions 
between process or learning approaches, and goal or objective driven ap- 
proaches. Within an open-systems framework in which learning and feed- 
back takes place it is also possible to have well defined guidelines or speci- 
fications which could be described as goals or possibly even blueprints. 
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JOSEPH HANLON 


The article on the front page of today’s newspaper or the first item on the 
radio news were probably written by journalists in less than a day. Even 
the longer ‘feature’ articles in newspapers and magazines are often written 
in less than a week. Journalists have developed techniques to ‘find out fast’ 
and present the results in a way that attracts attention. 


The problem of a policy investigator is often remarkably similar to that of 
the journalist. With limited background knowledge and little time, the in- 
vestigator must find an answer and present it convincingly. As a develop- 
ment journalist who is also a consultant and author I am using this chapter 
to share some of my tips or ‘tricks of a different trade’. 


The methods I suggest may not always be appropriate. They involve a lot 
of quick ‘thinking with paper’ (mainly grey materials —- compare 
O’Laughlin’s Chapter 5) and ‘thinking with people’ (mainly contacts — 
people who know about the issue already, ‘experts’or those already in- 
volved in the policy debate). They are useful in cases where someone com- 
missions me to answer a policy question directly: what’s the best way to 
provide funds for a project? How can we put pressure on government? (see 
Examples 3.1 and 3.2 and Section 3.6). You may need to answer a more 
specific field-based question: how do people make their living in a region? 
What is the impact of a new policy on poverty? My methods will probably 
be useful if you are investigating a policy question. If you are carrying out 
a more fundamental study, you may need some of the more formal meth- 
ods of ‘thinking with people’ (survey or PRA/RRA) discussed in Chapters 
6 and 7 or of ‘thinking with data’ (Chapters 11 and 12). Still, if your study 
is to be useful it has to link to a policy question, and the methods discussed 
here are probably useful as well. 


3.1 FOUR STEPS 


Because flexibility is central to journalistic investigations, there are no firm 
rules. But my investigative process commonly involves four steps: 


1 Throw the net out wider or ‘Find the woman who knows’. 


Journalists lack the time to reinvent the wheel, and our primary skill is 
making good use of people who have already thought about the issue. 
This means gathering existing reports and studies and talking, usually 
informally, to anyone who has investigated a related topic. In this phase, 
informal, unofficial, and ‘off-the-record’ contacts are key. It is essential 
in this phase to look more broadly than the narrow initial brief, throw- 
ing out a net to try to catch people and reports that look at the subject 
from different perspectives. This is essential because the journalist then 
must: 
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2 Redefine the question and select the key issue. 


Most investigators will admit, at least privately, that their first formula- 
tion of the question is normally wrong — often wildly so. Using the 
initial broader trawl of material and ideas, I decide on a priority issue as 
early as possible (certainly no later than the half-way point in an inves- 
tigation). At this point I make a first attempt to define the outcome — the 
broad outlines of the story or recommendations. This is the most prob- 
lematic point. 

The question needs to be redefined so that it is likely to provide both a 
useful answer and one that can be presented well. ‘Usefulness’ takes 
into account the assumptions, goals, and agendas of the people who 
commissioned the report or will read it. Both as a journalist and as a 
consultant, I already begin to think about presentation at this point. The 
best reports and articles make only one point, or at most very few. This 
is the time for discarding peripheral issues and selecting the one which 
will be the focus of the report or article. 


3 Discard and deepen. 


Having thrown the net out wider to draw in new ideas and material, it is 
vital not to be smothered in too much information. At this point, I ruth- 
lessly put aside all irrelevant material. A good journalist will discard at 
least two-thirds of the material initially collected. This is the opposite 
side of throwing out the net wider; it is selecting the relevant and dis- 
carding the irrelevant. What proves relevant is often surprising and dif- 
ferent than what I expected at the start — the whole point of throwing the 
net wider. 

This phase involves actually answering the newly defined question; 
numerous points must be clarified. This is the time to reread good re- 
ports and talk again to the best contacts, officially and ‘on the record’ if 
possible. 


4 Writing the report to have the desired result. 


Investigators need to be clear what action they want — is the report just 
for the file, or does the investigator want something done? The report 
needs to be written with that outcome in mind. Most people only read 
the summary or introduction, which becomes more important than the 
main report, and can be angled to push the desired result. 


Deadlines are the biggest difference between an academic investigation 
and a journalistic one. An academic doing a study for a PhD or a journal 
paper must cover a narrow area completely, often taking many years to 
complete the investigation. A journalist always has one eye on the dead- 
line, and investigates a broader area only to the depth that time allows; 
nevertheless, the result must be credible and accurate. Decision-makers, 
administrators and consultants are much closer to journalists than to aca- 
demics, which is why the journalist’s tools are useful. 
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3.2 STEP 1: FIND THE WOMAN WHO KNOWS 


Other people have already thought about the problem — whatever it is. The 
first rule of any journalist is to find the people who know. Initially, I do this 
on the telephone and informally, starting with friends and contacts and 
then quickly moving on to people met at meetings. I often make ‘cold’ 
calls, telephoning an agency and asking ‘Do you have someone who deals 
with X?". The telephone directory is the journalist’s most important refer- 
ence book. 


In talking to people, I find that admitting ignorance is always a good start, 
even when it isn’t true, because people are more willing to help when they 
feel superior and more knowledgeable. As well as asking people to talk 
broadly around the subject, I always ask two things: 


* if they know of any related reports, studies, articles etc. — both pub- 
lished and confidential — and 


* who else would it be useful to talk with (and I make sure to get a 
telephone number, at least). 


And I always take notes, even of an informal conversation. 


This sets up a telephone tree, with each person suggesting two or three 
others. In a few hours, it is possible to talk with a wide range of people 
about the issue. 


Often, there already exists a whole string of reports on the same subject or 
related ones, often commissioned from consultants and academics. In Chap- 
ter 5, Bridget O’ Laughlin talks of ‘grey material’; these are reports which 
are not published but which are often not secret either — or if they are 
officially confidential, they are still widely circulated. People are often 
pleased to share them — it is just that no-one asks and there is no central 
archive. Remember how few people will ever see your report, and how 
easy it would be for someone else to do the same thing again, without 
knowing what you did. For a book I was writing, I found 15 different stud- 
ies of poverty in Mozambique. 


Journalists are never ‘experts’ in anything; rather, our skill is in searching 
out the experts and in drawing on that expertise. At this stage, I give equal 
time and weight to people and paper. Hiding in an office and simply read- 
ing reports will never give an adequate picture. I want at least to skim as 
much as possible that has been written on the subject, but I also want to 
talk to as many people as possible who have already thought about the 
issue. 


There is the danger that particular views get repeated and gain credence 
that way with no-one ever checking to see if they are correct. However, 
reinventing the wheel is hard work and time consuming. The journalist’s 
key to ‘finding out fast’ is finding people who have already done the hard 
work, 
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Keep it unofficial 

There are three reasons to avoid official channels — it takes too long to get 
clearance, you are often forced to speak to the wrong person who does not 
actually know anything about the subject, and people are more cautious 
when they are speaking officially. If I want to use the material, I go back 
later officially. Informal chats on the telephone, or over lunch or a coffee, 
are always more productive in the initial stages of an investigation, be- 
cause they show how people are thinking on the subject. Like you, they 
will not commit to paper or official statements material about which they 
are unsure, or which goes against official policy, but they will be interested 
to discuss it informally. 


Confidentiality is essential. If I promise not to say you talked with me, I 
keep that promise at all cost. A record of keeping confidences helps in the 
future, because I can return to useful and talkative sources time and again. 
Recently, a government official referred an inquirer to one of my books, 
saying it was an accurate reflection of his views — even though he was 
never named in the book! 


Be sure of the rules. Journalists use the phrase ‘off the record’ to mean that 
what you are told can be printed, but you cannot say who told you. By 
contrast, ‘background’ means that you cannot use the material or identify 
the source — unless you can find someone else to tell you (which I can often 
do — once I know the basic story). Again, there are dangers here. Compare 
my suggestions with Bridget O’Laughlin’s more considered academic ap- 
proach in Chapter 5. In my opinion, although you should give references 
wherever possible in your final report, there is nothing to stop you using 
journalists’ pet phrases like ‘ministry sources say’ or ‘it is reported that’ if 
you need to protect a source. 


Sometimes possibly useful grey material is confidential or secret, often 
because reports are critical or came up with the ‘wrong’ answer. More com- 
monly, a person is afraid to release a report because they are not sure of its 
status. At this early stage, an investigator is more interested in the informa- 
tion and analysis, rather than quoting from the report or saying you have it. 
I use three techniques: 


* Ask for a copy in confidence, and respect that confidence. 


* Find out who else might have copies; someone more sympathetic may 
let me see it. 

* Ask to read the report. Often an official will let me sit in their office and 
read a report, even if they won’t release a copy. 


Finally, reciprocity is essential. If I ask people to talk with me unofficially, 
I must do the same with them when they need it. 


Using biased sources 
Openly biased people and groups can often be useful sources of background 
information. Pressure groups, trade unions, and others with an interest in 
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the outcome may have done extensive research to bolster their case, and 
that material can prove useful for background and for further sources. 
Groups with a strong agenda are also useful in obtaining questions to chal- 
lenge their opponents. 


In the study on land in Mozambique (Section 3.6 below), a key issue was 
whether land should be leasehold or freehold. Talking to strong proponents 
of both sides made clearer the implications of the differences in tenure, 
particularly with respect to loans and mortgages. Advocates of freehold 
argued that the only way peasants could obtain credit was through a mort- 
gage, and this had forced the leasehold advocates to do research on alterna- 
tive credit systems — saving me substantial work. 


Clearly care must be taken when using biased sources. But as I note later, 
and as Bridget O’ Laughlin notes in Chapter 5, all sources have some bias 
and the researcher must take this into account. The danger in ‘finding out 
fast’ is that the researcher will be fooled; every journalist has examples of 
apparently bona fide reports being false, with numbers changed or conclu- 
sions distorted. I often cite the case of the World Bank claiming in several 
widely distributed reports that a study of vegetable marketing showed that 
price deregulation had improved vegetable supply in Maputo — when the 
study itself said no such thing. 


Equally serious is the problem set out both by Chris Blackmore and Ray 
Ison in Chapter 2 and by Hazel Johnson and Linda Mayoux in Chapter 7, 
as well as by O’ Laughlin — power relations bias investigations. Outside 
investigators often misinterpret what relatively powerless people say — 
because they do not hear, or because people are telling them what they 
want to hear. This is linked to the issue set out by Sue Mayer in Chapter 13 
— that ‘expert knowledge’ often excludes key elements of ‘local knowl- 
edge’ because of the way the outside ‘experts’ codify and interpret. Partici- 
patory research techniques have been developed (see Chapters 6 and 7), 
but journalists do not have time to use them — we are dependent on studies 
already done by others. 


In assessing the validity of other's people’s work, I am guided by two things 
— my own sense of what seems reasonable, and the hope that I talk to 
enough people and see enough reports that some will challenge other dis- 
honest or distorted reports. In the end, though, we all make mistakes and 
misjudgements; the best I can ever do is keep them to a minimum. Step 3 
includes further attempts to reduce errors. 


Widening the net 

Telephone trees and collecting existing reports are two ways of throwing 
the net out further. In the terms of Blackmore and Ison’s Chapter 2, this can 
be seen as extending the ‘boundaries’ which define thinking on the subject. 
Often people have approached the same question from a different angle, 
and this will come up in informal discussions. For example, to draw to- 
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gether evaluations of hand pumps, it is not sufficient to look just to the 
rural water sector of government, but also to other development agencies 
that may have done project evaluations which, in passing, touched on prob- 
lems or successes with hand pumps. Informal chats on the telephone direct 
you to relevant reports. 


Widening the net is also about related questions. Starting out with a ques- 
tion of which hand pump is easiest to maintain leads to issues of spare 
parts supply and maintenance arrangements — is someone hired? is it done 
by an agency? is it done by a volunteer water committee? etc. Successful 
maintenance may prove to have more to do with organization and less to 
do with the pump itself. 


In a 1987 investigation of tightening sanctions against the then apartheid- 
ruled South Africa, a Commonwealth study team I headed obtained a study 
of United States sanctions against Cuba. This presented a different per- 
spective on the issue — the United States was trying to make sanctions 
work against Cuba when most countries were looking for mere token sanc- 
tions against South Africa. Thus it introduced alternative ways of thinking 
about what was possible and what was effective. 


The final point about throwing the net wider is time and speed. If someone 
is away on holiday or won’t talk to me, I stop and try someone else. If, with 
a bit of effort, you can’t get to see a report, forget it. You do need to be 
aware if there appears to be a whole area where people don’t want you to 
find out. Still, widening the net means drawing in more people and reports, 
which makes it easier to ignore those which are difficult to get access to. 


3.3 STEP 2: REDEFINE THE QUESTION 


Having thrown the net wider, the investigator then returns to rethink the 
question — and to look at those asking the question. It is essential to con- 
sider three issues: assumptions, agendas, and goals. As Examples 3.1 and 
3.2 show, these may only become clear during the investigation. 


The initial question is usually formulated on the basis of an implicit set of 
assumptions. Some are encapsulated in slogans such as ‘small is beauti- 
ful’. Others relate to the assumption that your employer — be it a govern- 
ment department, a bilateral donor, an NGO or a private company — is 
necessarily the best for the job. Some relate to skills and capacity — that 
foreigners or consultants are needed, or that peasants are too stupid. Key 
ones relate to the root of the problem, and whether it is technical, financial 
or organizational. It is worth listing as many of these assumptions as possi- 
ble; by making them explicit, they can be tested against the information 
drawn in through the earlier trawl. Note should be made of any assump- 
tions which do not obviously hold up. Example 3.1 shows a case of how 
this was done in a study of co-operatives in Zimbabwe. 
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EXAMPLE 3.1 ZIMBABWE CO-OPS: CHANGING THE QUESTION 


Several hundred producer co-operatives were formed in Zimbabwe 
in the first years after independence in 1980. Demobilized guerrillas 
and released political prisoners played an active role, and the govern- 
ment gave the movement political backing. Many co-operatives re- 
ceived technical and financial support from government and interna- 
tional NGOs. But by 1986 it was clear that many of the new co-ops 
were in trouble, and I was commissioned by a European NGO to doa 
study. I was to look at: 


(a) tensions between the government and the co-op movement; and 


(b) ways in which NGOs might provide further funding to co-ops 
‘without increasing the tensions with the government institutions’ . 


The underlying and stated assumption of the study was that the co- 
ops were in trouble because the government bureaucracy had turned 
hostile to them, despite political rhetoric, and that the solution was 
further funding by foreign NGOs in some way bypassing the govern- 
ment. 


In Zimbabwe, I talked informally to a wide range of people in the co- 
operative movement, in government, in the university, and in agen- 
cies supporting co-ops. I visited a number of co-ops — some success- 
ful, some struggling, and some failing. 


Informal chats and personal contacts confirmed two key assumptions: 
* most producer co-ops were, in fact, facing serious problems; and 


¢ there were sharp divisions within government and the civil service 
with respect to co-ops, and, indeed, key people who were sup- 
posed to be helping were in fact hostile to producer co-operatives. 


But I also found that government hostility was not the heart of the 
problem, and that in some cases the assumed solution — foreign NGO 
help — had actually been harmful to co-ops by creating dependency. 
Throwing the net wider made it clear that the question had to be rede- 
fined if my answer was to be useful to the co-ops. Further thinking 
showed that the core problem was attitudinal — foreign NGOs, gov- 
ernment officials, and many members of the less successful co-ops 
all treated co-ops as political, charitable and social organizations. On 
the other hand, the successful co-ops saw themselves as small busi- 
nesses which eventually had to be commercially viable. I decided 
that this had to be the key point of my report. 


Throwing the net wider also produced two surprises, even to me: 


* Idiscovered that in the six years of independence, some sectors of 
the largely white business community had significantly shifted their 


73 


CONCEPTUALIZING POLICY-RELATED INVESTIGATION 


attitudes toward the largely black co-operative movement. At least 
some commercial finance was now available, which my commis- 
sioning agency — and, indeed, many co-ops — did not realize. 

* TLalso found that the white colonial government had provided sub- 
stantial agricultural extension and other support to inexperienced 
new white farmers in the 1950s and 1960s, and that the new ma- 
jority rule government was expecting farming co-ops with a simi- 
lar lack of experience to thrive with much less support. 


My problem was that spreading the net wider challenged the core 
assumptions of many co-op members and — most importantly — of 
many international NGOs working in Zimbabwe. It became clear that 
two further unstated underlying assumptions were that co-ops were 
passive recipients of charity, and government and business were hos- 
tile. These, in turn, masked an agenda of many NGOs to continue to 
provide direct support to particular partner co-operatives, and not to 
encourage self-sufficiency or to build the co-op movement in gen- 
eral. Others wanted to build the co-op movement, but make it de- 
pendent on foreign funding rather than local business or government. 


Keeping in mind the sensibilities of all concerned, I reformulated the 
question to be: 


How can foreign NGOs help co-ops to become more politically and 
commercially independent and make the best use of government and 
private resources? 


Thus my proposals were to call for foreign NGOs to: 


* use their funds not to finance co-ops directly, as in the past, but to 
guarantee and support private commercial loans — thus setting up a 
sustainable system and routeing more money to the co-ops; 

© support co-ops in political lobbying and pressure on the govern- 
ment to provide more agricultural extension and other technical 
support, and help co-ops create their own technical support units. 


For some co-ops and NGOs who enjoyed the cosy direct relationship 
involved in cash handouts, these proposals were unacceptable; NGO 
workers would lose the power that comes from direct funding, while 
co-ops would have to be productive and profitable. 


But others found it useful. The report was one of many inputs that led 
to the establishment of a co-operative loan fund which had both com- 
mercial and NGO money. 
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Next is the question of agendas. Everyone involved in any project has per- 
sonal goals and agendas. When you do a study, you want to keep your job, 
or earn a promotion, or gain future contracts. I am writing this chapter not 
only to help you learn, but also to earn money and in the hope of winning 
more contracts from the Open University. Agendas are sometimes seen as 
shameful and illegitimate and thus to be kept hidden, because develop- 
ment agencies and charities are supposed to be altruistic and concerned 
only with the poor. But decision-making is easier when agendas are recog- 
nized and made explicit. 


In particular, this means making clear those agendas which may work against 
the long term development goals. Agendas of this type can be loosely 
grouped into four categories: 


1 Something to show. Development agencies need quick results to show 
to their paymasters — parliaments, national or international develop- 
ment agencies, or the general public making donations. This often in- 
volves photos of white women helping black children. 

2 Corporate stability. Many development agencies are large businesses 
with headquarters’ staffs and overheads to support; this is as true of 
non-government organizations as of ministries of health or agriculture. 
The agenda may be how to win a contract over another agency, in order 
to protect jobs or to expand. 

3 Power and importance. Everyone needs to feel that their role is impor- 
tant; every agency wants to believe that recipients of aid are better off 
for it. Individuals, departments and agencies often simply want to in- 
crease their power base. 


4 Other benefits. The highest priority for the Irish aid programme is ‘to 
increase the number of Irish people working in developing countries’ 
(Irish Department of Foreign Affairs, 1993). 


Finally, questions are often muddled because of lack of quantifiable objec- 
tives. Too often, goals are vague, such as ‘empowerment’ or ‘helping the 
poorest’. In their study Non-Governmental Organizations and Rural Pov- 
erty Alleviation Riddell and Robinson (1995) note that in the projects they 
studied objectives were ‘imprecise’ and frequently altered. This is both 
because NGOs wish to remain flexible, and because they don’t want 
quantifiable goals because ‘northern NGOs tend to exaggerate the poten- 
tial impact they can make’. 


But for rapid decision-making, it becomes essential to define a clear goal, 
even if one does not exist for the overall programme. Example 3.2 looks at 
the issue of goal definition. Goals must take into account both assumptions 
and agendas. 
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EXAMPLE 3.2 MATCHING A NEW QUESTION TO THE AGENDA 


Peasants in Namialo in northern Mozambique are poor and malnour- 
ished, so you are asked to propose an improved extension service. 
Throwing the net wider means looking not only at extension models, 
but also at peasant poverty. That immediately points to an assump- 
tion: that extension is the answer to peasant poverty. Namialo is a 
cotton producing area, and poverty and malnutrition are common in 
cotton outgrower schemes. This leads to three very different choices: 
1 increase food production (perhaps at the expense of cotton), for 
which extension is probably most appropriate; or 
2 raise the price paid to peasants for cotton, which is an issue of state 
setting of minimum prices; or 
3 improve the marketing structure to ensure that peasants can buy 
food at the lowest possible price, which may require state support 
for the private trading sector including loans and road building. 


Other more radical options range from promoting marketing co-op- 
eratives to providing better water supplies so that women spend less 
time collecting water and more time growing food. 


But assume your analysis of the agenda of the commissioning agency 
(an NGO, the Ministry of Agriculture, or a UN agency) is that it needs 
a rural extension scheme even if that is not the most productive, so 
your immediate goal is to work out the best extension scheme. But 
‘best’ for what? There are a range of possible goals: 

* reduce chronic malnutrition levels; 

* reduce acute malnutrition levels; 

* increase cash incomes of the poorest 20% of families; 

* increase women’s income; 

* increase total marketed production; 

* raise food production of the poorest in the village; 

* raise total food production in the village; 

¢ richer farmers providing more jobs to the poorest; 

* total production up; 

© productivity up. 

And so on. Which extension system is chosen depends strongly on 
which goal is chosen. 


Throwing the net wider also raises questions about the immediate 
objectives of extension itself. Is it to introduce better food crop vari- 
eties, to encourage best practice relating to existing crops and tech- 
niques, to introduce new forms of pest control, or to introduce en- 
tirely new forms of cash crops (in Namialo this might include oil 
seeds or tobacco)? 
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This last point raises another agenda issue — is cotton production to 
be defended or can cotton production be allowed to fall? Assume that 
because of the need to support private cotton companies and to en- 
courage exports, peasants must continue to grow cotton. 


With all of this in mind, the question might be redefined as follows: 
what would be the best form of agricultural extension to bring chronic 
malnutrition below 5% of children and to increase the food produc- 
tion of the poorest peasants without reducing cotton production? 


When goals do exist, they are too often short-term, such as number of 
wells dug rather than number still in use in three years, or numbers of 
health centres built rather than number of people using health facilities. A 
standard measure for the success of cancer treatments is five year survival 
rates — what percentage of those treated are still alive five years later. Simi- 
lar longer term goals are needed for development projects, but are rarely 
used, 


The new question 

New information almost always convinces me that the original question is 
not the best one. Initial research also makes clear the assumptions, goals 
and agendas that underlie the initial question. With these in mind, the ques- 
tion is redefined in a way to provide a useful answer and which can be 
presented well. 


Often it is worth making explicit more than one goal —a corporate goal and 
a development goal. Thus a simple question about the best hand pump 
might be turned into a new one: what is the best combination of pump and 
maintenance system which will give the donor short term publicity pic- 
tures and have the best chance of surviving for five years? Agendas are 
important too, so a question which started out being about rural health care 
might end up being one about making the best use of foreign volunteers in 
rural health posts. 


It is not too early to think about presentation. If I expect people to take 
notice (not always the case, see step 4), I make sure that my article or 
report makes only one central point, and perhaps a small number of sub- 
sidiary ones. Thus it becomes essential to define one key issue, and one or 
two points to be made about that issue. All the examples in this chapter 
show how this can be done. 


Having key points is not the same as having answers. For me, key points 
are usually clearly defined questions to answer. It is important to note that 
the new question may not be harder to answer than the original one — but 
the answer surely will be more useful and easier to present. 
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Trying not to get it wrong 

This phase is the most problematic and riskiest, because it requires choices. 
There is no single right choice, no single key issue or single right question. 
But there are wrong choices. In part, decision-making is intuitive and comes 
from experience. Usually a key issue will jump out at me during the initial 
investigation; the problem comes when there is more than one issue com- 
peting for attention, and I know I must choose one. 


A chat with a friend or colleague is useful at this point; even with a very 
tight magazine deadline that may only leave me a couple of hours, I find it 
useful to take two minutes to discuss the topic with someone. Indeed, be- 
ing forced to spell out the issue quickly often makes the decision for me — 
if I cannot explain it in a few words, then I probably cannot find out fast 
either. 


Expected outcome (step 4) is an important criterion. There is no point in a 
report which so contradicts agendas and assumptions that it is ignored. My 
Zimbabwe redefinition (Example 3.1) came very close to being wrong be- 
cause the new question was unacceptable to many donor NGOs. On the 
other hand my Mozambique redefinition, in the example in Section 3.6 
below, actually fitted better with the needs of all parties than the original 
question. 


A good journalist must be prepared to take risks. If I do not improve on the 
initial question, my articles will be too boring to be published. Any failures 
or mistakes will be outweighed by the successes. For a government offi- 
cial, the rewards for success may be lower and the penalties for failure 
higher. Each investigator must make their own choice. But I remain con- 
vinced that good investigation always involves some risk of failure — if 
you can be sure in advance, then there is no need for research. 


3.4 STEP 3: DISCARD AND DEEPEN 


Having redefined the question, you are now faced with a pile of reports 
and notes which are no longer relevant. Get rid of them. Now. 


It is hard to force yourself to think more broadly as you cast the net, but 
even harder to narrow down again and throw away what you do not need. 
At this point an unusual degree of ruthlessness is required — to exclude all 
issues not relevant to the main story. Since redefining the question is an 
iterative process, and you may redefine again, it’s as well to keep notes and 
reports you think you don’t need somewhere where you can retrieve them 
at least until this particular job is finished. Some will then be filed away for 
future use; others will be forgotten. But no matter how interesting, ‘finding 
out fast’ means keeping an eye on the deadline and only following the most 
important issues. 


I find this one of the most difficult steps. It hurts to discard reports or 
interviews that took hours of hard work or a long trip to obtain. I have to 
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admit that I cheat. Sometimes discarded reports sit in piles on my office 
floor for months because I cannot bear actually to throw them away. And 
on a longer project, if my initial investigations generate something that I 
find interesting but which is not directly relevant to the key issue, I some- 
times put it into the report as an appendix; I know no one will read it, but I 
gain pleasure writing it up. 


But as I have already noted, a journalist always keeps one eye on the dead- 
line, whether it is minutes or days away. When half of the available time is 
used, if I have not passed through steps one and two and moved on to 
defining the key issue and revised question, and being reasonably sure of 
the story, I am in trouble because I won’t meet my deadline. 


In part, this step revolves around answering a personal question: Why am I 
writing this report or article? It is essential to select the key point to be 
made, and structure the report — and the remainder of the investigation — 
around that. If you listen to a good radio news item or read a good news 
article, you will see they rarely make more than one key point and the rest 
of the article is to support that point. The best reports make only one point; 
never make more than a handful. Even if the report has dozens of conclu- 
sions, some must be more important than others, and need to be prioritized. 


Deepening understanding 

The task is now to answer the revised and much narrowed question. I never 
find the material already gathered and interviews already done to be suffi- 
cient. 


By this step, the deadline is in sight; time is precious and cannot be wasted 
following up all issues. Articles are written and decisions taken based on 
limited information; not even historians have all the facts. The goal is to 
collect as much as possible of the most relevant information. I make a list 
of remaining issues in order of importance to the final report, and start with 
the most important. Another journalist's trick is simply to leave out unan- 
swered questions or doubtful material that cannot be verified. Reports and 
articles are never ‘complete’ and my personal rule is: when in doubt, leave 
it out. 


This is where I go back to the most interesting reports and most useful 
contacts, to ask this very limited set of new questions. Having put the issue 
in context, clarified agendas and goals, and revised the question, I usually 
have new things to ask my most productive contacts. There is no harm in 
going back two or three times to the same person, when new research throws 
up new topics. Indeed, I find that when I go back a second or third time to 
people who have previously looked at these issues, they find my new in- 
formation interesting and there is a useful exchange of ideas. 


Having started out broadly and then chosen a narrow focus, this is the 
phase of deepening understanding. Unresolved issues need to be followed 
up. Conflicting opinions which were simply recorded in step 1 now need 
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to be clarified and either resolved or spelled out clearly. This is also the 
time to go back to pressure groups and others with overt biases, and put to 
them the arguments of the other side. 


This is also the final chance to prevent mistakes. If my conclusion rests on 
one or two key items, I try to verify their accuracy further — by looking in 
more detail at research methods, by talking to others who were there, or by 
talking again to people who disagree with the conclusions. Many of the 
other chapters in this book outline warning signals to watch for — is it pos- 
sible, for example, that ‘experts’ have missed a key point or that the re- 
ceived wisdom is wrong? Do the spokespeople of the intended beneficiar- 
ies of a project really speak for them? Even at this late stage, it is better to 
have second thoughts and delay than to go ahead with something if you 
have strong doubts. For a journalist, there is an intangible sense of ‘feel’ 
that comes from experience (and from having got things wrong before) — if 
the story ‘feels right’ I go ahead, but if nagging doubts make it ‘feel wrong’, 
it’s back to step 2. 


Finally, in this deepening process, I think much more about the final re- 
port. That means I try, where possible, to move ‘on the record’ so that I can 
quote people. It may now be useful to go through the proper channels and 
request formal permission to speak to people with whom I previously had 
an informal chat. Sometimes this is not possible and another informal chat 
is the only way forward. If confidence is promised, I always keep that 
promise. 


3.5 STEP 4: HAVING THE DESIRED RESULT 

Presentation is one-quarter of the job. Proper presentation improves the 
chances that the report will have the desired effect. Goals and agendas are 
important here, too, and we consider three cases. 

For the file 

Many reports are only written for the file ‘just in case’ — they are to prove 
that something was done or that warnings were properly considered. Many 
are done by consultants, but civil servants trying to protect their own jobs 
often do such reports. 


There is no harm in reports for the file, and presentation should be de- 
signed to discourage reading: 

© no summary or introduction; 

* use lots of jargon; and 

* bury the key points. 

To be read but not acted on 

Many reports are written to demonstrate good intentions, for example on 
gender equality, or to justify a decision already made. These reports are to 
be read by donors, funders, auditors, etc. in the hope that they accept the 
report and do not demand further action. 
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Most people only read the summary or introduction, and this should be 
encouraged by making the main report as dense and impenetrable as possi- 
ble. The introduction should take the form of an executive summary, which 
precisely follows the report section by section and simply summarizes what 
has been written. Be careful not to highlight key points; only suggest ac- 
tions that have already been agreed. 


Take action 

If action is desired in response to an investigation, then the entire report 
should be tailored to that end. The writer of the report must decide on the 
desired outcome before writing the report. Such outcomes need to be clearly 
and simply stated — a new kind of water committee, changing the form of 
agricultural extensions, support for the peasant union in rewriting the land 
law (see example in Section 3.6 below), a campaign against the World 
Bank, etc. It should be possible to state the desired outcome in one short 
sentence. 


Of course the investigation may lead to a whole series of subsidiary rec- 
ommendations, but it is essential to keep the main one to the fore. In writ- 
ing the report, recommendations need to be justified and key ones high- 
lighted. Debated issues must be discussed, but irrelevant issues should be 
dropped — even when you worked hard to find the information. To make 
the report easy to read by non-experts who may need to take decisions, 
jargon should be excised and replaced by plain English; acronyms should 
be shunned. 


Keep the report interesting. Journalists’ tricks like telling stories and quot- 
ing the more colourful speech of some sources can be used in reports just 
as well as in articles. Concrete examples and real names and places make 
reports and articles feel more real. 


If the investigator has a clear recommendation but there is a strong oppos- 
ing view, then the summary should only make the case for the investiga- 
tor’s view, while the main report should also clearly explain the opposing 
line. 


The introduction or summary is critical, because few will read the entire 
report. It should take the form of highlights and not a précis or exact sum- 
mary. Instead of a formal summary, I usually use an article pinpointing the 
key points and the action proposed. It must be short (only one or two pages) 
and, like any good article, it can only make one point — the action I want 
taken. 


There are two good ways to begin an introduction: 


1 Start with a strong quote or brief story that either sets the scene or backs 
up the course of action proposed. A few words from a government min- 
ister, a bishop, or a real peasant (with a name and village) proposing the 
course of action I want immediately makes the recommendation seem 
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reasonable and broadly acceptable. The alternative is to set the scene; a 
report on pumps might start: ‘Anna Mpofu had to walk 10 km each day 
to the river for water after the Chifunde village hand pump broke and 
no one could fix it’. Or 


2 Start with the key conclusion or single most important recommenda- 
tion, stated briefly and without caveats — I always leave the ‘ifs’ and 
‘buts’ for the main report. If I start with a quote or story, I give the key 
conclusion immediately afterwards. 


I would never start a report the way Open University books start their chap- 
ters, with the phrase ‘this chapter’ — only dedicated students read on. If you 
want people to read what you write, make it interesting from the very first 
words. 


The introduction/summary/highlights should never cover all points — that 
is the purpose of the main report. Instead, the introduction should make the 
case for the most important desired action. In any case, a good and pro- 
vocative introduction is more likely to push people into reading the whole 
report. 


No answers 

The fear of all investigators and journalists is of failing to find a clear 

answer. ‘Finding out fast’ does not always work. Journalists have two re- 

sponses to this. 

* Change the question again. Step 2 of the process is to redefine the ques- 
tion, and it needs to be defined in a way that has an answer. Go back and 
define it again. 

* Admit you don’t know. Journalists often use phrases like ‘opinion re- 
mains divided on ...’ or ‘the debate continues about ...’. 


The whole process of throwing the net wider encourages a reconsideration 
of the initial question and redefinition of the issues. If the investigator genu- 
inely remains unsure even at the end of the study, then a recommendation 
for more fundamental consideration is justified. Usually, however, simply 
proposing more study is a way of avoiding choices and delaying decisions. 
If that is not the goal (see step 2) then it is better to make a choice. 


3.6 STEP-BY-STEP: LAND IN MOZAMBIQUE 


‘Peasants are being pushed off their land in Mozambique and people are 
afraid to talk about it; tell us how to put public pressure on the government 
to stop this.’ 


That was the original request on the telephone from a British NGO. Find- 
ing out fast started even before accepting the job; a few telephone calls 
showed that some peasants were indeed being pushed off land, but others 
were defending their rights, and there was actually a massive public de- 
bate. With the brief rewritten to a broader one of helping to protect peasant 
land rights, I set off to Mozambique. 
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Step 1, throwing out the net, involved two days of intensive telephoning. 
As the telephone tree expanded, I found myself pursuing four different but 
overlapping lines: 
1 Identifying people already working on land rights — finding the people 
who know. I discovered that the government had already established a 
-National Land Commission which was working on just this issue, try- 
ing to draft revisions to the existing land law. (My commissioning NGO 
had not known of the existence of this commission.) Several 
Mozambican lawyers were also working on aspects of the problem and 
anxious to talk; women’s land rights had been raised in the Mozambican 
submission to the Beijing Women’s Conference in 1995. Two different 
groups at the local university were studying land issues. The issue had 
been raised by the parliamentary agriculture committee, and there had 
been substantial press coverage. 


2 Locating other studies. Land turned out to be fashionable with donors, 
in part because of a debate about whether privatization of land would 
lead to landlessness, as it had in Brazil. Thus there was a whole book- 
shelf of studies. 


3 The first few telephone calls produced three Mozambican organizations 
already working with peasants on land issues; during the next few days, 
four more Mozambican groups — including the Catholic Church and the 
national women’s organization — and three donor agencies all working 
on land rights were identified. Remarkably, no single organization knew 
about all the others in the field. 


4 Looking for first hand evidence of peasant land struggles, and their 
success or failure. What was striking was how successful peasants had 
been in defending their land. I found instances were someone came and 
claimed (often falsely) that the land belonged to them or a ‘minister’ 
and peasants left without protest (respect for authority remains strong). 
But I found no case where peasants with some right to land had been 
pushed off if they defended those rights. Indeed I discovered a variety 
of groups successfully gaining land titles for peasants. 


Step 2, redefining the question. The original brief was based on three false 
assumptions — that peasants were unable to defend their land, that the gov- 
ernment was uninterested in the problem, and that there was little public 
debate. I concluded that the government was sincere about getting a re- 
vised land law on the books by the end of 1996, that the National Land 
Commission was open to discussion and lobbying from outside agencies 
(and even had peasant representatives on it), and that in most instances 
peasants could defend their land (although the procedures were often cum- 
bersome and they needed assistance). The Commission had already ap- 
proved a national land policy under which peasants would confirm land 
rights through a mix of customary law and collective titles. My commis- 
sioning NGO had two relatively open agendas — supporting its partner 
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agency in Mozambique and gaining publicity at home — and one hidden 
one — keeping control in London and not yielding too much power to its 
Mozambican partner (which was one of those agencies already working 
with peasants, and which already had links with the Commission). The 
original vague goal had been simply defending peasant land. 


Based on initial research and the constraints defined by the agendas and 
vague goals, I defined a new question: How should the ongoing process of 
revising the land law be influenced so as to defend peasant rights, and how 
should the partner agency play a role in the process? My personal agenda 
was to shift power to Mozambique, and to counter the hidden agenda of 
the commissioning NGO to keep control. I kept at the back of my mind the 
need for publicity for the commissioning NGO. 


Step 3. First came discarding and error avoidance. Many of the studies on 
land tenure systems and arcane debates about Mozambican law could now 
be put aside. A check against an important possible error was to return to 
lawyers and others to confirm my view that many peasants had success- 
fully defended their land rights. 


Next my focus was on deepening my understanding of the ongoing politi- 
cal and legislative process and how to influence it. Further discussions 
with the Land Commission and its technical advisors, with peasant pres- 
sure groups already identified, and with the most interesting lawyers and 
researchers highlighted two sides of the ‘how’ question — what mecha- 
nisms should be used, and which areas should be targeted. 


It was clear that peasants needed a louder and clearer voice in the ongoing 
process, and to do this I concluded that some form of lobbying was needed. 
This would work directly with the Land Commission and feed into the 
ongoing press debate. 


The other part was more complex. It seemed that the mix of customary and 
collective titles could be made to work, and further research highlighted 
key issues that needed to be dealt with in the land law. These related to how 
decisions were taken, transparency, rights of women, and dispute settle- 
ments. 


Step 4, having the desired result. My agenda was that action should be 
taken, so my report was aimed in that way. Having two recommendations 
and accepting my own rule of only making one point, I concluded that the 
key point for my paymasters in London was that they should support peas- 
ant lobbying, and my introduction highlighted this. It also stressed ways in 
which original assumptions were false and how the Land Commission was 
doing a good job. The debates over customary law and key issues in the 
land law were relegated to the main report. 


The second point, about key issues that the land law needed to consider, 
was more for Mozambican consumption, so I wrote an article for a local 
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newspaper setting out those recommendations. To satisfy publicity demands, 
I also wrote two general articles for the British press, which named the 
commissioning NGO. 


In summary, we see that the initial question, despite its false assumptions, 
hid a real question which could be usefully answered to the benefit of all 
parties. 


3.7 RISKS AND BENEFITS 


For a journalist, the point about ‘finding out fast’ is to meet the deadline 
and publish an article. For the investigator, the point is to make a re- 
commendation of some sort. 


Journalists are notorious for making mistakes. Pressure of time means we 
do get things wrong. I have written stories where I believed people who 
lied to me and where I quoted dishonest or inaccurate reports. Conclusions 
based on limited information have been wrong. I have made a bad choice, 
missing the main issue and following up a less important point. 


But if you check press reports against your own experience, you will find 
the press is far more often right than wrong. These techniques do work. 
Perhaps most important, taking no decision at all is usually worse than 
making a wrong choice. Quick decision-making moves development for- 
ward. 


Go ahead. Take the risk. 
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PART II 
THINKING WITH PAPER 


4 HOWTO DO A LITERATURE STUDY 


STEPHANIE BARRIENTOS 


Any kind of research or investigation in policy is going to involve, at some 
level, a study of the relevant literature. If you are the investigator yourself, 
this study may be the main focus of your work, or it may simply serve as 
background for other forms of research. Either way, you should be able to 
analyse and use the literature in order to support your own research project. 
It is likely this is going to involve you in a search for information using 
literature from a diverse range of sources, be they academic, non-govern- 
mental organizations (NGOs), governmental or community groups. You 
may well need to combine your use of literature with other sources of in- 
formation such as interviews, surveys or data, and all may form part of any 
final report you present. However, a literature study can also constitute a 
policy investigation in its own right, if enough evidence can be obtained 
from analysing what others have written. 


It is useful to distinguish the idea of a literature study from that of a litera- 
ture survey. A literature survey involves you primarily in summarizing the 
material as accurately as possible. A literature study involves a process of 
critically assessing and evaluating the same material in order to develop 
your own analytical approach and/or relate it to new aspects of inquiry. 
While a literature study is more focused, a literature survey is generally a 
necessary preliminary phase in any such study. 


In this chapter we will examine the aims of a literature study, the methods 
by which a literature study can be carried out, the interface between litera- 
ture and other sources of information, and finally, how you can use litera- 
ture to formulate your own argument and present your own analysis. It is 
important that you learn to evaluate the literature you read in a critical way, 
and develop your own arguments and structure so that your final written 
work stands up as a coherent piece of analysis. Even if you are in a position 
of commissioning and using the results of others’ research, rather than do- 
ing the investigation yourself, the same skills will be useful. On the one 
hand you need to understand what the researchers have to do and the diffi- 
culties involved; on the other you have to evaluate their reports critically 
yourself — and those reports are themselves contributions to the literature! 


It is important to stress that literature-based research is a constantly inter- 
related process of: 


(a) gathering and assimilating; 
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(b) evaluating and analysing; 
(c) formulating your own arguments on the basis of your analysis of what 
you have gathered; 


(d) structuring and writing up your arguments. 


Although you will be focusing more on gathering/assimilating informa- 
tion at the beginning and evaluating/analysing later, they are not completely 
separate or discrete stages. Throughout we will consider the problems of 
time constraint. You will need to structure and plan your research in order 
to progress and achieve results within a realistic time-frame, and in this 
respect it is important to divide your research into different, if overlapping, 
stages. 


The chapter has four main sections, related to the different stages you will 
need to go through in order to complete your own literature study. At the 
same time the sections are interconnected, discussing the process you are 
likely to go through in your research. You may find some sections more 
relevant than others, depending on your own purposes in using the litera- 
ture. The main focus of this chapter is on literature-based research. But 
even if you are only using literature as a background to other types of 
research — rapid rural appraisal (RRA), participatory rural appraisal (PRA) 
or data analysis, for example — you still might find it useful to think about 
all aspects of literature-based research, as in your final outcome you will 
probably be integrating results from your primary and secondary data 
sources with information gathered from the literature. 


This chapter mainly deals with published or accessible literature, and should 
be read in conjunction with Chapter 5 by Bridget O’ Laughlin, which con- 
siders many of the same issues using ‘grey material’ and institutional lit- 
erature. 


Section 4.1 considers aspects of the literature search, possible sources that 
can be used, and problems of timing and information availability. It then 
considers the related issues of surveying the literature you acquire, and 
using this to trace further sources and to help focus your own study so that 
it is manageable and leads to productive results. This focusing involves 
posing a central question, which will guide your study throughout. 


Section 4.2 examines how you can approach studying the literature you 
read. Having already posed the main question of your study, you can ana- 
lyse the available literature in that light. This is broken into three parts: (a) 
how to evaluate an argument or approach; (b) how to locate an argument or 
approach; and (c) how to compare different arguments or approaches. There 
are no hard and fast rules for these, but this section attempts to give you 
guidelines, and indicates problems you might encounter. 


Section 4.3 discusses the next stage, which is the process of formulating 
your own argument. The step from summarizing other people’s ideas to 
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articulating your own can be a very difficult one. Again there are no hard 
and fast rules, but once again posing the right question is central to this 
process. Your analysis of the literature was related to the central question 
you posed. Now you draw your own conclusions from this analysis which 
forms the basis of your own argument. It is this process which is essential 
to completing a coherent, well-argued study based on the literature. 


Section 4.4 examines the final stage of structuring your study, linking all 
the material you include to the central argument, and writing it up in a way 
which is readily accessible to the reader. If you have been successful in 
formulating your own argument, this stage of your study should fall into 
place fairly easily. 


In this chapter, I will develop and use a single example of a literature-based 
study throughout. The example I use is part of a study examining ‘Policies 
towards women temporary workers in Chilean agribusiness’. This example 
study is based on similar research that has been carried out in Chile — 
Barrientos (1997), Valdes (1992) and Venegas (1993). Of course, your own 
study could be completely different from this example, but it provides con- 
crete indications of the types of problems you might encounter. 


While this chapter can give you insights into the process of literature-based 
research, ultimately there is no single formula or set of hard and fast rules. 
You will need to use your own initiative, logic and intuition to achieve the 
results you are aiming for. The main thing is that, whatever your topic, you 
consider it from the different aspects this chapter explores, even if you 
arrive at the conclusion that certain angles are more relevant to your spe- 
cific research design than others. Research can be a bumpy road, but hope- 
fully this chapter will help you to think objectively about the process you 
are involved in, and help you avoid some of the potholes — thereby making 
the journey all the more worthwhile. 


4.1 LITERATURE SEARCH AND SURVEY 


This initial phase of a literature study is intended to be wide ranging to help 
you acquire a broad understanding of the topic, and then progressively nar- 
rows down to focus on a more manageable area to survey. The aim is to 
define the central question for the literature study that will guide the later 
analytical phases. 


4.1.1 LITERATURE SEARCH 

It is likely that once you, as a development manager, initiate your investi- 
gation of a topic, you will be operating under tight constraints, both in 
terms of the time you have available and your physical access to sources of 
information. If you have little knowledge of the topic, this is going to be 
more problematic than if it relates to a similar area you have already re- 
searched. When starting from scratch, your first aim should be to try to find 
literature which gives an overview of the topic: any articles, books or pam- 
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phlets which survey or summarize the field. This is a quick way of famil- 
iarizing yourself with the area of the project; getting to know what work 
has already been done (there is no point in reinventing the wheel) and 
giving you key sources or references. You might find these by contacting 
key institutions or organizations, visiting relevant bookshops or libraries, 
or by talking to someone who has already done related research. 


A common problem is that the best sources of published information are 
often located far away from the main sources of primary information. If 
you are researching in a developing country, you may find that the best 
publications tend to be in one or more of the developed countries (which 
have the main publishers and libraries and many institutional head- 
quarters), while the main primary sources will of course be in the develop- 
ing country itself. Ideally you will be able to visit both, but if not, try and 
find contacts who might lend you information, and write to people or 
organizations abroad to try and obtain information you are lacking (do this 
early on as it will take time). Ultimately your research will be constrained 
by the information you have access to, and you should remember this in 
determining the focus of your study. 


Wherever you are based while doing the work, you will hopefully be able 
to visit specialist libraries. Most large international organizations have re- 
gional libraries, and many government and non-government organizations 
have sources of information in both published and unpublished form. 
O’Laughlin deals with the latter type of material in depth, and from the 
beginning you should gather as much relevant information as possible from 
all sources (the most frustrating thing in a literature search is not picking 
up something you had access to which you later find you need). You will 
need to visit or write off to publishers, research institutes, journals, or even 
authors themselves. If you have access to a computer, use the Internet as a 
source of information, and make contact through e-mail. Even if you have 
access to a library, you should also contact other possible sources of infor- 
mation, which do not necessarily log their publications in libraries, such as 
NGOs, banks, international institutions, government departments, or other 
bodies relevant to your field. You will need to allow time for this. 


If you are carrying out your work in a developed country or large city, it is 
important to visit any relevant academic or public library which is access- 
ible. Most UK academic libraries now have computerized cataloguing and 
search facilities. You will hopefully be able to gain access to search facili- 
ties, such as LIBERTAS, NISS and BIDS, as well as to use CD-ROMs to 
search for sources in your specific area of interest. Ask librarians for help. 
If they become interested in your quest, they can give important leads. 


When you use formal or computerized search facilities you are going to 
have to start defining the focus of your study by the key words you use to 
initiate your computerized literature search. Equally, you must try differ- 
ent key words and combinations to be sure you are accessing all the 
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relevant literature available. This is shown using my example study in 
Example 4.1. Not only should you put in obvious key words and as many 
related combinations of key words as possible, but also try and think of 
alternative words for the topic you are searching, and don’t forget to search 
for material related to particular institutions as well. 


EXAMPLE 4.1 FINDING KEY WORDS 


Example study title: ‘Policies towards women temporary workers in 
Chilean agribusiness’. 

Obvious key words: ‘women’, ‘Chile’ and ‘agribusiness’. 

Other key words which might also help to access relevant literature: 
*gender’, ‘Latin America’, ‘agriculture’, ‘rural’, ‘employment’, ‘gov- 
ernment’, ‘state’, ‘politics’ etc. 


Alternative key words: ‘horticulture’, ‘food’, ‘work’, ‘peasant’. 


If you are using good search facilities, you will be getting immediate feed- 
back on the amount of literature available, and hence on research already 
completed, which might also give you different angles your research could 
possibly focus on. Initially, it is useful to look for fairly general books, 
journal articles, or reports in your area. This gives you an overview of your 
field before you narrow your topic down. But you will need to define your 
topic as soon as you can so that you use your time productively. How you 
narrow your topic may be largely influenced by your brief and your or- 
ganization’s policy requirements, but your own particular interests will play 
a part, as well as the availability of material, resources and the time you 
have available. It is very easy, through your interest and enthusiasm in a 
subject, to be overambitious in your aims. However, any research project 
has to be of manageable size and have clearly stated objectives if it is to be 
successful in fulfilling those objectives. 


Having defined your topic, you must now start pursuing references and 
leads in the literature more avidly. Keep a note of all relevant references 
you come across that might be of interest, as other authors’ references and 
bibliographies are effectively ready-made partial literature searches. Also, 
where you see certain works being referred to repeatedly by different 
authors, you are getting an indication of the main works and debates in the 
field of your study. 


You will probably continue your literature search, although less avidly, 
quite far on into the research for your study, as you continue to encounter 
and chase references you had not seen before, or decide to explore new 
angles you had not initially considered. But gradually its importance will 
decline as you move further into your study, and your own work becomes 
more focused. 
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4.1.2 LITERATURE SURVEY 

As you accumulate and read material, so you embark on your literature 
survey. When you first start surveying the literature, you do so fairly ‘blind’, 
taking in much of what you read as it is presented. You might have noted 
different ideas and approaches in the literature, and certainly papers, arti- 
cles and books will have examined the topic from different angles. 


Initially this will be a fairly broad survey, but as you narrow and define 
your topic, so you will begin to concentrate your survey on the literature of 
most relevant interest to you. As soon as you have defined your topic, you 
need to write a brief plan of your study, which acts as a guide for your 
further search, survey and analysis of the literature. It is very easy when 
you are doing a literature project to read ad infinitum, but you could well 
be wasting time which you do not have. A plan helps you to focus and also 
to allocate your time appropriately. This early plan is not definitive, can 
easily be amended, and can be integrated into your terms of reference if it 
is part of a broader research project involving, for example, field study. 


4.1.3 FOCUS: INITIAL PLAN AND CENTRAL QUESTION 

The initial plan should give you a central focus and framework for your 
study. You achieve this by identifying the specific aim of your project — 
which is the central question you want to address. Each section of your 
project should then relate to this central aim, which acts as a backbone to 
your project. We are going to explore this in some detail later on, as this is 
an essential part of carrying out a literature project. But the earlier you 
identify this central aim, the more you are able to focus your research 
productively. As a general guide this initial plan should be as brief as poss- 
ible (one side of A4 paper is enough). It should contain the following 
information: 

* Title 

* Aim (problem and central question you are going to address) 

¢ Key issues (with short summary of each) 

© Other research methods to be used (summary of methodology) 

* Time schedule 


This outline acts as a ‘photograph’ of your project which you can keep in 
mind at all times. It helps you to organize your material, discard literature 
which is not relevant to your specific needs, concentrate on the sections of 
any book or article of greatest importance, and search for new relevant 
material. At the same time you should write a time schedule for your work, 
detailing your time allocation for: 

* literature research; 

* other research methods (interviews, RRA and PRA, data analysis, etc.); 
* writing, allowing time for redrafting and polishing at the end. 


This helps you to organize and divide your research time proportionately 
between the parts of your project, within a manageable deadline. 
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4.1.4 LITERATURE STUDY 

As you progress with your literature survey, so you will need to begin to 
study the literature analytically. In Chapter 5, O’ Laughlin writes of mov- 
ing from a survey to a study of the literature. Here I prefer the idea of 
search/survey as the first phase in a literature study, followed by analysis. 
However, there is no single point of transition from a survey to the later 
phases of a literature study. As you begin to analyse, you will continue to 
search and survey new work, guided in part by the ideas you are beginning 
to form. But increasingly you should be able to read from a more critical 
standpoint and be able to relate it to any additional research (field work, 
survey material or data analysis) you are going to use. 


When you critically assess material, you do not do so in a vacuum. An 
essential ingredient in this is that, as early as possible, you will have posed 
a question which you want to address, and from then on you approach the 
literature in the light of analysing its relevance to your question. Having a 
central question is important both in giving focus to your reading and in 
framing any further research design. It will play a central role through to 
the final conclusion of your investigation. In Example 4.2 I consider how 
this might affect my example study. 


EXAMPLE 4.2 POSING A QUESTION 


In my example study, I started with a fairly broad title ‘Policies to- 
wards women temporary workers in Chilean agribusiness’. But this 
is still fairly general, and could involve uncritically summarizing a 
range of policies, including those of NGOs and the government, with- 
out any yardstick for comparing or assessing them. 


Posing a central question helps to resolve this, and the question I will 
pose is: ‘Which policies help to empower women temporary work- 
ers?’. This question can now help to inform the literature search. 

* ‘Empowerment’ needs to be entered as a keyword in the computer 
search. In general the search should explore literature on the em- 
powerment of women (Kabeer, 1994). 

* It helps to focus analytical reading of the literature. In any book, 
article or document, I am interested in the information given, but 
now I particularly want to ask how this affects the empowerment 
of the women themselves. 

¢ It helps to evaluate critically the literature used. Even if an article 
doesn’t mention empowerment, what are its implications? 

* It helps to interpret and unpack any given policy and data. 

* It poses a framework for carrying out further in-depth research 
such as semi-structured or focus group interviews with temporary 
women workers, and helps the design of that research. 

* Ultimately, it will provide the basis for writing up the research. 


93 


THINKING WITH PAPER 


4.2 ANALYSING THE LITERATURE 


We now need to consider how to analyse the literature we are surveying. 
One problem you are likely to face in development or policy studies is that 
the literature may be from a range of different sources: academic, govern- 
ment, non-government or other organizations. They are going to be writ- 
ten from different perspectives, in different styles, using different ‘in-house’ 
language. O’Laughlin examines the use of grey and institutional material 
in Chapter 5, but let’s consider the general issues. First we'll look at how 
to assess articles or reports individually, before going on to comparing 
them. 


4.2.1 EVALUATING AN ARGUMENT 


The process of critically assessing a piece of analysis starts with evaluat- 
ing its position or argument and logical construction. Evaluating a position 
or argument can be broken down into two important parts. Firstly, you 
have to identify the argument accurately. If you misunderstand or misrep- 
resent an argument, your own final analysis of it will be flawed. Secondly, 
you have to consider whether the argument you are examining is logically 
consistent internally and meets its own objectives. In other words, do the 
conclusions follow from the premises? We will consider each of these sepa- 
rately. 


Identifying an argument 

The extent to which identifying an argument is easy or problematic will 
depend largely on the author you are reading. Some authors present their 
ideas very clearly and systematically in an articulate form. They will start 
at the beginning with a summary of their views, the questions they wish to 
address, the material they are going to examine, the methods they are go- 
ing to employ and the conclusions they aim to draw. So long as they fulfil 
their objectives clearly, then your main role is to take accurate notes. 


Often, though, written work appears very complex, and cannot be easily 
summarized in a few paragraphs. In this case you have to be very patient in 
working through the argument, accurately recording all the essential points, 
in order to get an overall view of the main features and their interconnec- 
tions. Sometimes, a piece of writing looks highly complex, but once you 
have isolated and identified its essential points, it is actually quite simple. 
Do not be intimidated by the fact that some writers have perfected the art 
of dressing up mundane ideas in sophisticated language. Therefore, you 
need to question everything you read, but at the same time be accurate in 
distinguishing and recording the main features of its argument. Remem- 
ber, too, the points Joseph Hanlon made in Chapter 3 about writing to 
impress. He was thinking about how you might improve your writing style 
and write with different aims in mind, but of course similar considerations 
also applied to others writing up the reports which are now part of the 
literature to you. 


94 


4 HOWTO DOA LITERATURE STUDY 


Alternatively, you might encounter an empirical piece of work containing 
a large amount of factual information and data, but no clear argument. The 
way data are selected, organized and presented often reflects a particular 
perspective, whether or not that perspective is explicitly stated. In this situ- 
ation you need to stand back from the piece of work, and consider what 
type of data have been considered relevant and how these data have been 
selected and are organized. You might also need to consider what data or 
evidence have not been included. Often you can evaluate a piece of litera- 
ture just as much by what has not been considered, as by what has (see 
Mayer, Chapter 13). 


The other problem you might encounter is when, however hard you try, 
you are unable to identify exactly what the author is trying to say. It is not 
easy to give advice on this, except to suggest that (a) you try to get as close 
an approximation to the argument as possible, and (b) if you decide to use 
the argument in your own final report, you indicate that this was the argu- 
ment as you understood it. 


The logical consistency of an argument 

Examining a piece of work for logical consistency will depend partly on its 
structure. In turn, the structure might vary, in accordance with the method- 
ology employed by that author, with the particular approach or institu- 
tional perspective of that author, and with the nature of the work under 
examination. The main thing to look for in determining logical consist- 
ency is whether the conclusion logically follows from the premise, given 
the assumptions made, and is supported by the evidence given. 


In determining the internal consistency of a piece of work, you must be 
aware of the criteria that piece of work sets itself. Does it pose a central 
question or issue that it is going to address? Does each stage in the argu- 
ment or procedure adopted follow logically from the previous one? Do the 
data and empirical evidence support the argument, or are there contradic- 
tions? Are there dislocations in the argument? Does it arrive at a conclu- 
sion consistent with its original aim and supported by the evidence given? 
I am trying to provoke you, when reading, to ask yourself whether you 
think a piece is consistent, or whether it contains contradictions in terms of 
its own argument or approach which undermine its validity as a piece of 
analysis. Example 4.3 considers the type of literature encountered in my 
example study, and the anomalies which might arise. 


What we have been considering so far is the evaluation of a piece of work 
in its own terms. The next stage is to locate each argument in relation to 
others, before going on to a comparison between one argument and an- 
other, and weighing them up in relation to available evidence (data, sur- 
veys, etc.) which has not necessarily been included. Finally you will need 
to consider the importance of an argument for your own study. 
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EXAMPLE 4.3 EVALUATING THE LITERATURE ON WOMEN IN 
CHILEAN AGRIBUSINESS 


Let us consider this process of identification and evaluation using my 
example study. Assume that the literature I have encountered through 
my search and survey can be broadly divided into three categories: 


1 Chilean agribusiness. Much of the literature is easy to identify 
and categorize. Most of the work on agribusiness extols the suc- 
cess of Chilean agribusiness in global markets, and argues that 
low labour costs (no mention of gender) have been a key element 
in achieving Chile’s comparative advantage, creating new job op- 
portunities. 

2 Women and empowerment. At the other end of the spectrum, some 
literature on women and empowerment emphasizes the high de- 
gree of exploitation, poor work conditions and insecurity of the 
women drawn into temporary work in agribusiness, emphasizing 
the disempowering effect on women. 


3 Government and non-government policies. The literature relating 
to policy, coming from a diverse range of institutions, might be 
more difficult to identify (see O’Laughlin, Chapter 5, for further 
discussion on this). 


Anomalies 

However, there are some important anomalies. I might encounter an 
article extolling the benefits of agroexports, which argues it is neces- 
sary to maintain low labour costs, to improve labour productivity and 
remain competitive in global markets. In this case, I might question 
whether it is logically consistent, and analytically supported, that cheap 
labour is the best mechanism for improving labour productivity. 


Alternatively, I might read another article which examines the way 
rural women have been drawn out of necessity away from traditional 
agriculture into working in agribusiness, with disempowering conse- 
quences. But here I might question whether traditional agriculture 
was really that much more empowering for women, and thus whether 
the argument over the disempowering effect of agribusiness has been 
logically supported. 


lam not at this stage agreeing or disagreeing with the position of the 
article. What I am examining is whether the article is internally con- 
sistent, and is able to support the argument or analysis it advocates. 


4.2.2 LOCATING AN ARGUMENT 


Locating a particular report or article in relation to others means being able 
to place it in the context from which it arose and also to identify where it 
fits in terms of different analytical approaches. 
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1 Examining the context 


It is likely that the literature you are going to compare comes from a 
number of different time periods, institutional sources, theoretical ap- 
proaches, ideological perspectives, each reflected in different policy 
agendas. However, none of it was written in a vacuum, and examining 
the context can provide important insights into a better understanding 
of it. The greater your understanding of the problems different authors 
have sought to address, and how those problems arose, the greater your 
understanding of their analytical development and any conflicts between 
the different approaches you encounter. 


The extent to which you need to look at the context will partly be dic- 
tated by the type of topic you are researching. In most policy-based 
research, you may need to identify the development process and the 
specific problems which arose in order to contextualize the different 
policy proposals, and also consider the position of the different actors 
within any given situation. This is particularly important in comparing 
work from different agencies, in which you need to consider the institu- 
tional context, whose interests are being represented, and what is the 
underlying policy agenda (see Chapter 5). 

Exploring the context of a topic can involve consulting a wide range of 
literature on economic history, social and political and policy related is- 
sues, both in relation to the country or countries and to the topic you are 
examining. This can be very time consuming. If you decide upon this course, 
you will only be able to consult a sample of sources; try to look in the 
literature for any overviews, literature surveys, and summaries already car- 
ried out by other authors. You might also find it helpful, for example, to see 
if similar work to your own has been carried out for another country, from 
which you might gain contextual insights. You will need to use your own 
initiative on this, but you must be very careful to limit the time you spend, 
and stick rigorously to your original time schedule. 


2 Different analytical approaches 


Development and any other field of policy studies are also composed of 
a number of quite different approaches, which you must also be aware 
of in evaluating the literature, contextualizing your project, and devel- 
oping your own analysis (Hunt, 1989; Toye, 1993). You will not have 
time to go back and familiarize yourself with each different approach 
(unless you are specifically examining one of these in your study), and 
will have to fall back on previous knowledge. But you might well have 
to refer back, and fill in gaps, in order to attempt to place any piece of 
literature in terms of its theoretical or analytical approach. 

Whenever you read an article, book or piece of analysis, it is important 
to try to locate it in terms of which approach it reflects. When an author 
indicates clearly the viewpoint he or she is starting from, this is quite 
easy, but many authors will not render you this service, or might be 
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combining ideas from different schools. There are four aspects you need 
to concentrate on when attempting to identify the approach of a particu- 
lar author, although the relative importance of each will vary according 
to the type of literature you are reading. 


© What are the key theoretical assumptions? 
© What empirical evidence or data are used? 
© What are the policy prescriptions? 

© What references are cited? 


Let us assume that you are looking at two pieces of work on structural 
adjustment and poverty written from neo-liberal and basic needs perspec- 
tives, for example. If the author assumes free markets and free trade are the 
most efficient method of development, cites key macro-economic indica- 
tors, argues that greater economic liberalization is the most beneficial policy 
prescription, and cites authors such as Krueger (1974), the work is of neo- 
liberal origin. If, conversely, the author starts with an analysis of entitle- 
ments and capabilities, appeals to social, poverty and equity indicators, 
and advocates community-based social policy as an essential element in 
resolving problems of economic development, citing authors such as Cornia, 
Jolly and Stewart (1987), it is likely the work comes from a basic needs 
perspective (Hunt, 1989; Toye, 1993; Stewart, 1995). 


These, however, are two fairly clear-cut examples. The problem of locat- 
ing a piece of writing in terms of a particular approach can be difficult. A 
piece of writing may contain a number of competing influences in explain- 
ing a particular phenomenon. Once you extend this to policy analysis, which 
by necessity involves a degree of pragmatism, pinning a proposal to a spe- 
cific analytical approach can be even more difficult. Therefore, it is up to 
you to gauge the main influences and general perspective of any work, and 
weigh up the relative importance of its theoretical perspective to the given 
literature you are studying. 


How you do this again will depend very much on the type of topic you are 
researching, but for example you could develop a chart to group the litera- 
ture you are using, and if necessary break it into component parts accord- 
ing to its leaning. Let us assume, for example, your study involves using 
material from two approaches (neo-liberal and basic needs), and you want 
to examine the background, analytical, empirical and policy angles of the 
topic. Much of the literature you are dealing with might not neatly fit into 
one approach or another, so a chart of the key aspects you are examining 
could help you break down an individual article, and help as a method of 
comparison of the literature. 


A possible example of such a chart is given in Table 4.1. Using this you 
could note (or tick) the leaning of each component part of an article, use 
this to weigh up its overall approach and compare it with other articles 
similarly deconstructed. This is a fairly mechanical method. You would 
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need to evolve your own version of such a method with which you feel 
comfortable, and which would help you to think logically through the proc- 
ess of evaluation and comparison. 


TABLE 4.1 A SIMPLE BREAKDOWN OF THE LITERATURE 


Neo-liberal Basic needs 


4.2.3, COMPARING ARGUMENTS 

The next stage in a literature project is when you begin to compare and 
weigh up arguments, which may be from different theoretical perspectives. 
There are no hard and fast rules for comparing different arguments. Hope- 
fully, you will begin to do this intuitively as you are reading, but it is im- 
portant that you are aware of this as an essential transition to developing 
your own arguments and conclusions. 


The problem of comparison really lies with the literature which is not easy to 
group by approach. For example, two articles might appear to agree in the 
main about analysis of a problem, but come to quite different or opposing 
conclusions in terms of policy. Here you will need to question each article as 
we have above. What context were they written in? What are their unstated 
theoretical assumptions? What are their institutional agendas? What evidence 
do they appeal to? What authors do they cite? What are the unstated implica- 
tions of their policy prescriptions? All of these types of questions will help you 
to unpack and weigh up the literature. But ultimately, you will have to use 
your own initiative in comparing this type of literature. 


However, one additional test, which is vital, and leads on to the next stage, 
is: How do any two pieces of literature relate to the central question you 
are asking? At this stage you are not simply comparing the literature ‘blind’ 
or in a general way. You are weighing it up and comparing it in preparation 
for your own use, i.e. in order to formulate your own argument or to inte- 
grate it with other forms of research which your question is leading to. 
Therefore, analysing and probing it in terms of your own central question 
is a way of comparing literature, not in and of itself, but in terms of its 
relative importance to the project you yourself are carrying out. To what 
extent do two articles relate to your question? To what extent do they throw 
new light on your question (whether or not you agree with them)? To what 
extent do they refute or support the available evidence and data? What 
further research do they necessitate? How do they relate to the arguments 
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you are developing, and conclusion you are now beginning to arrive at? 
You are now involving yourself, and your own research project, in the 
process of comparison, and at the same time, using the literature as part of 
the process of formulating your own arguments and further analysis. It is 
this process which ultimately constitutes the advancement of research. 


Example 4.4 uses my example study to consider the problem of evaluating 
the literature in more detail, and the type of chart which might help us to 
break down the literature into its component parts for the purpose of com- 
parison. : 


EXAMPLE 4.4 COMPARING THE LITERATURE 


My example study deals with issues relating to agribusiness, govern- 
ment and non-government policy, and women’s needs. I have also 
formulated my question regarding the empowerment of temporary 
women workers, which I want to use in analysing the literature for 
my own purposes. I thus need to elaborate a chart as a means of break- 
ing down and evaluating the literature I am studying. Table 4.2 gives 
a simple example of a chart I could use. 


Any literature which clearly supports a free-market, free-trade ap- 
proach to agroexports and completely ignores the problems of women 
workers will clearly be grouped in the first of each pair of subcolumns: 
‘Pro’ agribusiness, ‘Negative biased’ on gender, and ‘Gender 
disempowering’ on policy. 


Conversely, any literature which is critical of the effect of agribusiness, 
and which emphasizes the problems faced by women temporary work- 
ers and the importance of nursery provision, improved pay/working 
conditions, transport, health provision, social provision and educa- 
tion for temporary workers, will clearly be grouped in the second of 
the subcolumns: ‘Critical’ of agribusiness, ‘Positive perspective’ on 
gender, and ‘Gender empowering’ on policy. 


The problem is going to be in locating literature — for example, policy 
documents coming from government organizations — which advo- 
cates a combination of the above. For example, such documents might 
be critical of agribusiness but ignore the problems faced by tempo- 
rary women workers. Conversely, they might support agribusiness 
but also argue that women workers need more support. My central 
question provides an important litmus test for comparing different 
pieces of this type. 


Analysis of the existing literature also reveals a lack of input from the 
women temporary workers themselves on this question. I therefore 
need to set up semi-structured interviews in order to gain more infor- 
mation on the impact of different policies and programmes on gender 
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relations in the work place and household, and the views of women 
on their relative effects. 


But finally I will need to use my own analytical and methodological 
skills to assess and compare the different literature in relation to my 
analysis of the interviews. 


TABLE 4.2 


Leaning of literature 
Agribusiness Gender analysis Policy 
Area of Pro Critical Negative Positive Gender Gender 


literature biased perspective disempowering empowering 
——— ee ee 
Agribusiness 

Gender 

Community 

policy 

Government 

policy 


4.2.4 MOVING THE LITERATURE BOUNDARY 

So far we have concentrated on examining literature in a published form 
(quite likely including pamphlets and institutional literature). But we must 
remember that all published literature has a ‘vintage’, be it new or old. The 
information contained in a recently published book, for example, could be 
at least one to two years old, allowing for the process of writing and print- 
ing etc. (although some non-academic publishers can speed up the proc- 
ess). The information contained in an article or pamphlet might be more 
recent, but once published, it has entered the public domain and there is 
little point in simply reiterating it. 


An examination of the existing literature is always an essential part of 
research, but it is likely you will need to move beyond this boundary if you 
are to produce material of additional value. Within the context of a litera- 
ture project itself, this could involve trying to find unpublished material 
(see Chapter 5), or material which is still in the process of being devel- 
oped, by attending relevant talks or conferences. Through this you become 
familiar with similar work which is in progress (again no point in simply 
reiterating it), or gain access to new sources of information such as discus- 
sion or policy documents. 


The main way of gaining access to this type of supplementary material is 
to contact other people, be they key informants, professional experts, or 
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other researchers (see Hanlon, Chapter 3). It is usually beneficial before 
talking to such people to have done your own groundwork first, so that you 
can hope to have an informed discussion with them, and understand the 
context and background to any new information. But you can also see some 
of the problems and issues other researchers in the field are grappling with, 
which can help inform the process you are going through. Research is of- 
ten an isolated process, and simply talking to someone who is knowledge- 
able in the field can help you to formulate your own ideas. If you are work- 
ing in a developing country, with little library access, this could also be a 
means of access to published literature you cannot easily obtain. 


Moving the literature boundary is also the point of interaction with non- 
literature forms of research. Other chapters explore these in detail, but you 
should always remember that a literature project can contextualize, inform 
and help to direct, and literature is ultimately a means of communicating 
the results of other research mediums. So, for example, you may want to 
supplement your literature project with semi-structured interviews, or a 
survey in the field. In both, the literature will have helped you to examine 
the general issues, and to identify weak areas which have not been ad- 
dressed. This will help you in the formulation of your questions, and the 
design of any questionnaire (Mikkelsen, 1995). You might choose to pur- 
sue the data you have encountered in the literature. At the simplest level 
you may go back and analyse the original data source, or you may have 
obtained new secondary or primary data you wish to analyse in order to 
supplement, advance or question the existing literature in the field 
(Mukherjee and Wuyts, Chapter 11, and Mayer, Chapter 13, discuss these 
issues in more detail). 


Alternatively, your primary aim may not be a literature project, but rapid 
or participatory rural appraisal (see Johnson and Mayoux, Chapter 7) or an 
econometric project, which the literature simply helps to inform. Again, 
the literature can be a means of contextualizing the research agenda, or 
getting a handle on the data or evidence you are analysing (Section 4.3 
examines this), and help you to confirm or interpret the results you obtain. 
The research process often involves constant interaction between quantita- 
tive and qualitative analysis at different levels (micro, meso and macro), 
especially in innovative policy analysis. Finally, your main aim may be to 
undertake case study analysis (see Thomas, Chapter 14), where literature 
provides one of several forms of evidence. 


4.3 FORMULATING YOUR OWN ARGUMENTS 


Once your research is well under way, you will need to start formulating 
your own argument or analysis of the topic. This can be a very difficult part 
of a literature project. Summarizing other people’s ideas can be fairly sim- 
ple, but formulating your own involves a lot of energy and critical thought. 
Don’t worry if you find it difficult; the best ideas will not necessarily come 
easily. There are three aspects to formulating your own argument: 
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1 the central question you want to ask; 
2 the approach you want to take; 
3 the conclusion you want to draw. 


I will examine each of these in turn so that you can consider the role each 
plays in formulating your own logical argument. 


Central question 

We have already seen the importance of posing a central question for your 
analysis of the literature, and this now becomes a key element in formulat- 
ing your own argument. However, it is worth standing back for a moment 
and thinking through whether your question as originally posed is quite the 
right question. You may want to modify it a little or even reconceptualize 
around a rather different question that makes it easier to formulate your 
own argument. 


The central question can act as the ‘pivot’ on which you hang your own 
argument, and provides the backbone for articulating your own written 
work. Having a central question does not exclude also having a number of 
subsidiary questions, but they should all be connected to the central ques- 
tion, and it is this which will provide the backbone of the project, allowing 
it to stand as an integrated whole (see Section 4.4 below). 


Argument 
It is possible that when you started the project, you knew the argument you 


wanted to develop. If you are working for a particular institution or organi- 
zation, this might have been given to you as part of your original brief. If 
not, identifying a question can also help you to decide your argument. 


In literature-based policy analysis, you will often be dealing with debates. 
You will need to develop and articulate your own argument through explo- 
ration and critical examination of the existing approaches you consider. In 
doing this, you might decide to align yourself with one particular approach 
on the issue, although you should always consider alternatives and possi- 
ble criticisms of the stance you take. Alternatively, you might decide dif- 
ferent approaches have valid contributions to make to understanding the 
problem you have chosen to project. Whichever line of argument you de- 
cide upon, it is important that the analysis of it is clear and consistent and 
leads ultimately to your final conclusion. Remember that others will be 
analysing what you write, using the kind of critical approach put forward 
in the previous section! 


Conclusion 

For your own argument to be logical and coherent, you also have to arrive 
at an endpoint in this process. Your argument is the logical series of steps 
you have gone through — in your analysis of the existing literature and 
supporting evidence or research — which link the original question to the 
final answer. The argument you develop must be rigorous and the conclu- 
sion you draw should be logically consistent with your original question 
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and supported by the evidence you present. Your argument has also got to 
be able to stand up to criticism. The argument you develop should be ten- 
able, either in terms of logical reasoning or appeal to empirical data, and 
should consider potential refutations or criticisms. And each stage of your 
argument should be connected (however indirectly) and lead towards your 
final conclusion. Once all these elements are in place, you are in a position 
to write a well-structured and coherent report, which we will consider in 
more detail in the next section. 


4.4 STRUCTURING YOUR ARGUMENT AND WRITING UP 
YOUR REPORT 

Early on in your study you should have written a short outline of its struc- 
ture. As your research progresses, this outline must be expanded into a 
fuller structure through which you can develop your argument. The more 
coherent the argument you have elaborated, the more easily this structure 
should fall into place. The difference between the outline and this more 
elaborate structure is that you are now developing the complexities of the 
argument, and integrating into it the more peripheral points, while still 
ensuring that they are connected to the central question. Some points may 
not be relevant to the final argument, so as Hanlon suggests in Chapter 3 
you should be prepared to leave them out, even if they represent a lot of 
reading on your part. 


The final written report must stand as an interconnected whole. Each sec- 
tion should clearly be linked to the central question of your study, and 
those connections should be logical, as shown in the integrated literature 
study framework in Figure 4.1. Further, the progression from one section 
to another should be determined by and encompass the overall progression 
of the central argument of your project from its initial question to its final 
conclusion. Hence each section is connected to the whole like a skeleton to 
the backbone, which in our case is the central question. This skeletal form 
is summarized in Figure 4.1. 


central question section 1 

background 

section 2 

subsidiary analysis/conclusion 


section 3 
subsidiary analysis/conclusion 


section 4 
subsidiary analysis/conclusion 


central 
conclusion 


FIGURE 4.1 Literature study framework 
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If you already have a well-formulated argument, the structure of your overall 
study will probably fall into place fairly easily. But conversely, it is often 
through developing the structure and beginning to write your report that 
you ‘test’ the coherence of your argument, and you might find you need to 
refine it. The aim of the final structure, therefore, is to achieve the overall 
balance between sustaining the central argument of the study, while simul- 
taneously exploring its various facets through analysis of the relevant 
literature and supporting evidence. Example 4.5 gives a concrete.example 
of this, using my example study. 


EXAMPLE 4.5 LITERATURE STU: 
ARGUMENT AND STR 


E - FORMULATING AN 
URING THE REPORT 


Returning again to my example study, I had already decided the cen- 
tral question: ‘Which policies help the empowerment of women tem- 
porary workers?’ and this question has helped in the process of fo- 
cusing on, projecting and comparing the literature. This question is 
then used as the basis for formulating my own argument, which frames 
the structure of the final report itself. 


Policy on women temporary workers in Chilean agribusiness 
1 Introduction 


Brief outline of project and statement of aims: to examine prob- 
lems women temporary workers face; to examine policies towards 
them by state, NGOs, and community groups; to consider whether 
policies help to empower women temporary workers themselves. 
2 Background 

Considers the background and expansion of temporary female em- 
ployment in agribusiness. Defines empowerment and examines lit- 
erature on whether agribusiness has had an empowering or 
disempowering effect on women through increasing paid employ- 
ment. 


3 Policies to help temporary women workers 


Examines the main policies which have been implemented by gov- 
ernment and non-government organizations. Critically assesses 
them in terms of the extent to which they have enabled the greater 
empowerment of women. Integrates results of semi-structured in- 
terviews. Argues policies have been varied and not well co- 
ordinated, with mixed effects on empowerment of women. 


4 Joint programmes and community-based centres 


Examines evolution of community-based centres, and grass roots 
action. Considers the recent joint programmes (state, NGO, 
employer and trade union), aimed at setting up more community 
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centres for temporary workers. Integrates results of semi-structured 
interviews, and assesses the potential effect of community-based 
policies on women’s empowerment. 


5 Conclusion 


The conclusion, based on the analysis undertaken, is that the em- 
powering effect of agribusiness has been contradictory, and that 
only those policies which enhance the participation of women them- 
selves have had an empowering effect. The joint programmes and 
community-based centres contain possible problems of co-ordina- 
tion, but they help to meet practical gender needs, and if the col- 
lective participation and influence of women themselves can be 
ensured, they also have potential for addressing strategic gender 
needs. 


In this example, the formulation of the central question regarding empower- 
ment helps to link the analysis in each section, and thus the articulation of 
the argument which leads to the conclusion. The above structure provides 
the basis for writing the final report. It is only when you come to write that 
you will finally be sure whether you can sustain your central argument 
throughout, and you are able to tie all the separate parts of your analysis 
into an integrated whole. Always allow time for preliminary drafting, as it 
is bound to be fairly rough and disjointed at the beginning, and will need 
polishing to produce the final integrated whole. 


4.5 CONCLUDING REMARKS 

In this chapter, I have attempted to guide you through the various stages a 
literature study is likely to involve, and the types of problems you will 
need to address at these different stages. The aspect of this I have concen- 
trated on is how to transform a broad summary of the literature into an 
analytical study through which you can develop your own argument and 
policy analysis. 


A problem with this is that literature studies can vary widely in both the 
types of issue they are addressing, and how they address them. I have at- 
tempted solely to give you some guiding principles. It will be your final 
decision to choose what is relevant to you, given the topic you have been 
assigned or have chosen, and the approach you want to take. The main 
thing is that you approach your study with your eyes open, and an analyti- 
cal mind, that you question what you read, and question what you do, and 
that you use your own experience and initiative to achieve your objectives. 
The result will be a report that is consistent and clearly argued, and with 
which you can be satisfied as a piece of your own work and a useful contri- 
bution to policy. 
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In policy-oriented research, we rarely choose our own topic; we are as- 
signed it. In the course of our research, however, we often come to chal- 
lenge the terms or boundaries of the topic we were given. We construct our 
own account, and then decide how to present it to the policy makers or 
activists who commisioned the research, or how to reach a wider audience. 
The production of policy documents, our reading of them and the con- 
struction of our own accounts are shaped by the politics of the policy- 
making process. What policy-oriented research tells us is not self-evident; 
it has to be interpreted politically to be understood. 


In this chapter we will be looking at how, in the process of constructing 
and presenting our own account, we can get the most out of grey materials 
— unpublished documents which are available from various organizations 
such as governmental or non-governmental agencies. We will be looking 
particularly at policy-oriented reports on development issues written by 
trained researchers working either as employees or consultants for govern- 
mental or non-governmental organizations (NGOs). The examples used 
here derive mainly from my own experience as a researcher on develop- 
ment issues in two different contexts: first in a research and teaching cen- 
tre within the national university in Mozambique, then in a development 
institute in the Netherlands, looking at research on poverty in Africa. The 
issues raised here apply, however, to using grey material from other re- 
gions and on other topics as well. 


This chapter should be read as a complement to Chapter 4 on literature 
study. Whereas Stephanie Barrientos stressed the importance of locating 
an author within a particular school, here I am emphasizing the importance 
of locating research reports within particular institutional discourses. Re- 
search reports are often written anonymously so even their identity is insti- 
tutional. But even when signed, such reports must be located institution- 
ally. They are the product of an explicitly political process — though often 
couched in technical language. The final report is the outcome of a process 
of negotiation between researchers and the commissioning institution. To 
use this material well, we must locate the choice of questions and conclu- 
sions within this political process. We must recognize key words and inter- 
pret both messages and silences as part of socially constructed institutional 
discourses. 


5.1 PROBLEMS IN USING GREY MATERIALS AS SOURCES OF 


INFORMATION 


The policy making process that produces grey materials is an explicitly 
political process. This can compromise their usefulness as sources of in- 
formation in some persistent ways. The first is censorship — either explicit 
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or implicit. Agencies that request reports can simply refuse to accept con- 
clusions they do not like, assigning drafts to the dustbin or negotiating 
bland and uncontroversial conclusions. The World Bank, for example, disa- 
greeing with an outside evaluation of its own assessments of poverty pro- 
grammes in Africa, simply withheld it. An informative report on rural credit 
carried out by a Mozambican research team headed by an outside researcher 
circulated only clandestinely for years because it raised the question of 
smallholder credit at a time when the Frelimo party expected peasant (small- 
holder) farming to disappear. 


To avoid censorship, researchers often exercise self-censorship, since they 
wish both to have their research reported, and to be commissioned to do 
further research. In practice it is very difficult to write a definitively nega- 
tive report on any on-going project supported by those who commissioned 
the research. Motives may be political as well as financial. In researching 
agrarian issues during Mozambique’s socialist period, for example, we 
deliberately put little emphasis on class differentiation of the peasantry; 
we were concerned that evidence of differentiation would be used by cer- 
tain voices in the ruling party as a justification for forced collectivization. 
There is thus a drift toward consensus in grey material, often at the cost of 
coherence. The conclusions drawn and executive summaries do not al- 
ways reflect the findings and information presented in the body of the re- 
port. 


Not only conclusions, but even the general premises of research may be 
subject to implicit assumptions about suitable solutions. Ferguson (1990) 
has noted that development institutions have preconceptions about accept- 
able policy options for rural Africa. There is a standard package of projects 
— usually emphasizing greater integration of traditional peasants into the 
market — that they are prepared to implement across a wide range of situa- 
tions. He also argues that these packages tend to be technical — removed 
from the different political situations in which they must operate. In the 
case of Lesotho, an independent country located in the heart of South Af- 
rica and long a source of migrant workers, this package included technical 
measures such as rural road construction and improved livestock produc- 
tion. This was intended to improve commercial smallholder production. 
Ferguson argues that for this package to look like a solution to Lesotho’s 
problems, World Bank research on Lesotho had to rewrite its history and 
political economy to present it as ‘a nation of farmers, not wage laborers, a 
country with a geography, but not history, with people, but no classes; val- 
ues, but no structures; administrators, but no rulers; bureaucracy, but not 
politics’ (Ferguson, 1990, p. 66). The migrant workers moving constantly 
back and forth to South Africa thus became traditional peasant farmers, 
isolated from the market. 


Preconceptions about solutions may also explicitly figure in negotiation of 
research questions between researchers and those who commissioned the 
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research. During my time with the research centre in Mozambique we were 
asked to investigate worker absenteeism in the harbour of Maputo, consid- 
ered by management to be the principal cause for the long turn-around 
time in ship-handling. Preliminary research convinced us that low produc- 
tivity in the port was rooted in the organization of the work rather than in 
the recruitment of the workers. It took hard bargaining to extend the terms 
of our research to the operational zones of the harbour. 


The focus of much grey material is on ‘problems’ and this in itself leads to 
some predictable silences, even without political censorship. Neither space 
nor time is sufficient to locate the problems within their relevant historical 
and social context. These contexts are often assumed to be understood by 
both researchers and readers, sometimes, as Ferguson (1990) points out, 
most inappropriately so. The immediate political context that made certain 
points fundamental may no longer exist, and thus make the concerns of a 
particular report unintelligible or even trivial to the reader. For example, in 
reports produced by the research centre on co-operatization in Mozam- 
bique, we included what may now seem to be disproportionate detail on 
the ownership and cultivation of cashew trees. This was because forced 
resettlement in communal villages was still an option preferred by some 
within the ruling party. We wanted to make it clear why disputes around 
ownership of cashew trees had become a symptomatic focus of opposition 
to resettlement. 


The politics of the policy process leads to a dichotomy in the types of grey 
material it produces: (a) reports with a national focus, of the sort that min- 
istries and large international agencies tend to commission; and (b) project 
reports, commissioned by NGOs wanting to know how their money was 
spent, often focused on a particular committee or region and concerned 
with issues such as the competence of individual project staff. The critical 
analytical space in between these two types of reports — the links between 
macro or sectoral policies conceived at a national level and their disparate 
effects at local and regional levels — is rarely covered in grey material. 


Finally, the unequal power relations that dominate the policy process in 
turn dominate the production of grey materials. There is a substantial body 
of literature on participatory action research, much of which is explicitly 
concerned with the empowerment of the dispossessed through the research 
process itself. The number of studies realized through this process is not 
great, however, and their institutional presence tends to be ephemeral. The 
voices of the poor, the oppressed, and the exploited are under-represented 
in grey material, and the voices of the powerful echo loudly. In their re- 
view of studies of malnutrition in India, for example, Kumar and Stewart 
(1992, p. 278) observe that the discussion on targeting has tended to focus 
exclusively on leakage of benefits beyond the target group, ignoring fail- 
ure to reach all in the targeted group. They note that narrowly targeted 
interventions often show apparently favourable cost-benefit ratios, of great 
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importance to officials, but this does not tell us the costs incurred by leav- 
ing out groups in need of nutritional supplementation. The voices of the 
latter group are not so easily heard as those of officials concerned with 
showing the efficiency of their work. 


The dominance of the discourse of international consultancy in grey mate- 
rials on development issues reflects unequal power relations between coun- 
tries that are the focus of development studies and the countries that com- 
mission such studies. Problems of reliability that arise in the politics of the 
policy-making process are exacerbated by the prevalence of internation- 
ally recruited short-term consultants in the preparation of grey material. A 
lot of this work is financed by donor countries that consider international 
consultants better trained and less likely to defend local interests. Moreo- 
ver, local researchers do not always have facilities for turning out appro- 
priately formatted reports in acceptably smooth English or French, and 
they are less fluent in the institutional jargon and fashions of the moment. 
For example, in discussing possible research on the consequences of sharp 
cutbacks in the recruitment of Mozambican miners in South Africa with a 
staff member of the United Nations Development Programme (UNDP), 
researchers from the Centre for African Studies tried to explain that the 
principal problem was not unemployed miners, but rather the young rural 
unemployed men in Mozambique who needed wage-income but had never 
been employed in the mines. The staff member from UNDP understood 
the point, but explained that unemployed miners were a ‘glamourous’ is- 
sue, whereas rural unemployment was not. 


Short-term external consultants are prone to characteristic defects in re- 
search practice, the quick and dirty methods described by Chambers (1980), 
which this book is designed to counter. They rely for background informa- 
tion on previous consultancy reports, often furnished by the commission- 
ing agency, so that the same stereotypes are passed on from one report to 
another, acquiring the appearance of truth, the well-known fact, with each 
telling. They tend to concentrate their attention on centres of power, speak- 
ing with those who can rapidly mobilize facts and information: ministry 
officials, embassy desk officers, technical staff of international agencies. 
While we can learn from the best of such consultants (see Chapter 3), they 
all too often play the role of Simmel’s stranger — the outsider brought in to 
hear contradictory and conflicting accounts, order them, and present back 
to the people in power what they themselves said in a more orderly and 
acceptable way. Such consultants may also try to find sources of hard quan- 
titative data which will give an appearance of substance to their reports, 
although the methodologies used to collect these data are often illusively 
presented. The methodological basis of qualitative information is typically 
even less rigorous. Short-term consultants tend to rely on gossip and anec- 
dotes acquired in the corridors of the capital and in semi-social encounters 
with other international personnel. 
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5.2 WHY USE THIS MATERIAL AT ALL? 


If there are so many problems of interpretation in institutional discourses, 
why would we want to use grey material for finding out fast? Why not just 
put it aside and look exclusively at measured and critical academic re- 
search on our topic? 


The most important reason for finding and interpreting grey material is that it 
is itself an important part of the world that we are studying. It tells us the ways 
in which important institutions in the politics of development, such as the 
World Bank, UNICEF, COSATU (the South African trade union federation), 
and national governments view problems and solutions in the domain we are 
studying. When a poverty study on Mozambique informs us that ‘the pros- 
pects for off-farm employment in Mozambique as in much of Africa have 
never been very bright’ (World Bank, 1989, p. 102), we learn that current World 
Bank thinking on Mozambique ignores the long history of migrant labour to 
impose the general model of smallholder agricultural development which is 
current dogma on the alleviation of rural poverty in Africa. As long as we 
remember that what we learn from such studies is not the truth but the repre- 
sentation of particular points of view, they can be ideal key informants in the 
sense used by Chambers (1980): knowledgeable, interested, anxious to in- 
form — and open to interpretation. 


Analysing such reports as part of a political process clarifies the context of 
policy formulation. This in turn helps us to think about how we want to 
write up or present our own study. We can sound out the audience and 
determine what is likely to be assumed, what is likely to be contested and 
what is politically acceptable to different institutional actors in the policy 
context within which we are working. Current documents on restructuring 
the mining industry in South Africa, for example, all emphasize the impor- 
tance of raising productivity. But whereas reports done for the mine work- 
ers union, the NUM, emphasize greater stability and equality of employ- 
ment as ways of improving labour productivity, research published by the 
Chamber of Mines, the owners cartel, sees high labour costs as the princi- 
pal problem of the industry. Knowing both positions would help us to read 
the African National Congress (ANC) government’s Green Paper on re- 
structuring and to locate its strengths and silences. 


Grey materials can also be a source of information about the topic we are 
studying which further research of academic literature cannot provide. 
Consultants often have access to recent statistical information, registered 
either by governments or by the agencies for which they work, which is 
not in the public domain. Their interventions must be timely, thus reports 
are finished much more rapidly than are academic studies. Consultants 
may also work in situations where conventional academic research is diffi- 
cult or unauthorized. For many years, for example, information on 
Mozambican refugees came mainly through people working with agencies 
involved in refugee work. 


WW 


THINKING WITH PAPER: 


Finally, there is a high moral and political cost to be paid for repeated 
research on the same topic and in the same areas, particularly when this 
involves surveys, even though we may be able to detect many methodo- 
logical and conceptual problems in previous studies. Research is always 
an approximation, never a reproduction of reality. Critically reading, cross- 
checking, and recasting a series of fragmented, incomplete or even biased 
reports on our topic can contribute to giving us a reasonable understanding 
of the policy area we are studying. 


5.3 WHERE TO BEGIN? NOT WITH GREY MATERIAL 


How then do we undertake this critical reading? As Barrientos (Chapter 4) 
observes, the way we read and take notes depends on the phase of our 
research. Initially, to find the topic, or to develop alternative approaches to 
the topic we have been assigned, we survey the literature. The literature 
survey makes it possible both to define the topic more clearly and to iden- 
tify material which is worth reading closely and analytically in the litera- 
ture study as a basis for developing our own arguments. In both survey and 
study phases, grey material is not the place to begin. Just as we would 
never try to conduct a key interview without first having thought carefully 
about the topic and identified the social and political position of the person 
we are going to interview, we must also prepare ourselves before reading 
grey material. General historical information about the place we are study- 
ing and similar studies from other areas sharpen our awareness of alterna- 
tive theoretical and political approaches to the topic under investigation. 


It is easier to spot assumptions and incomplete arguments if we already 
know something about the history, geography, and political economy of 
the region under study. It was easy for Ferguson (1990) to recognize the 
absurdity in the Lesotho Country Economic Report (World Bank, 1975) of 
characterizing the peasantry as traditional subsistence cultivators, because 
he had read work such as Murray (1981) that traced the consequences of a 
long history of labour migration and the decline of marketed peasant pro- 
duction. 


Reading key sources on the issues and areas under study also makes it 
possible to recognize and evaluate the original sources of hand-me-down 
data. In Mozambique, Wuyts (1981) used data from colonial agricultural 
censuses to estimate regional differences in the proportion of peasant pro- 
duction that was marketed, thus making it possible to estimate total peas- 
ant production. Ten years later, these estimated proportions had become 
facts, and were used to estimate the impact of structural adjustment on 
peasant production. Total peasant production was inferred from registered 
marketed surpluses, producing a remarkable increase in estimated GDP. If 
one has read the original work by Wuyts and is also broadly familiar with 
the history and political economy of Mozambique, one doubts that esti- 
mates based on conditions of production and marketing in the 1960s could 
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be carried over to the late 1980s in the midst of a destructive war and after 
prolonged scarcity of consumer goods in rural markets. 


5.4 IDENTIFYING GREY MATERIAL FOR INTERPRETIVE 
ANALYSIS 

Chapter 4 shows you how to approach the literature in different ways in 
different phases of your research. In the literature survey you are still de- 
fining your topic and locating the directions of future study. In the litera- 
ture study you are developing your own argument. Since grey material is 
just a sub-set of the literature in policy-oriented research (though a particu- 
larly important and distinctive kind), you should approach it in the same 
way. However, I will distinguish between two ways of reading — the struc- 
tured skim, and a more thorough analytical and interpretive reading. Each 
can be necessary in both literature survey and literature study. The deeper 
interpretive reading always depends on the structured skim having first 
identified what is useful for study. 


5.4.1 FINDING POSSIBLY USEFUL GREY MATERIAL 

The first problem to be solved in identifying grey material for finding out 
fast is obtaining access to it. The second is separating the wheat from the 
chaff. The World Bank has improved public access to some of its reports; a 
number of which can even be down loaded from the Internet. Many na- 
tional planning commissions have also relaxed their hold on strategic eco- 
nomic information. None the less, it is still very difficult to obtain most 
current research reports, particularly if findings or topics are controversial. 
Consultancy firms usually do not distribute their reports widely, as these 
are the only product they have to sell. Government officials continue to be 
suspicious of researchers and reluctant to share previous reports, particu- 
larly those that are critical. 


Finding out that a report exists is usually the first step to obtaining access. 
Locally based researchers often know about useful grey material even if, 
ironically, they sometimes have less access to it than outside consultants. 
Some grey materials are covered by bibliographic data bases, such as Cur- 
rent Contents. The World Bank home page on the Internet gives access to 
lists of its publicly available reports and DEVLINE covers grey material 
catalogued by the library of the Institute for Development Studies (IDS) in 
Sussex, United Kingdom. The local World Bank offices and NGOs keep 
copies of reports they have commissioned and often have access to gov- 
ernment reports that are not in the public domain. Copies of consultancy 
reports can sometimes be obtained from their clients, particularly the de- 
velopment-oriented NGOs. The United Nations Development Programme 
(UNDP), United States Agency for International Development (USAID) 
and central planning ministries generally have documentation centres for 
the use of their employees and consultants. Specialized government agen- 
cies are sometimes willing to allow researchers access to material in re- 
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tired archives, though reluctant to release current studies for research they 
do not control. 


5.4.2 SORTING THE WHEAT FROM THE CHAFF: THE STRUCTURED SKIM 
READING 

After obtaining access to grey material, the second major problem is de- 
ciding what is really useful and well worth reading in our literature study, 
and putting aside the rest, much of which is weighty but useless. The ab- 
stracting services provide good help for deciding whether a report is worth 
looking for, but they are not usually a sufficiently reliable guide to their 
content. A structured skim reading helps us sort out the material we have 
accumulated. It is particularly important for grey material, because some- 
times it turns out to be our only reading — we may not have another oppor- 
tunity to look at a report. Government or agency officials may allow you to 
look at reports but not to take them out of the office or perhaps not even to 
photocopy them. On a return visit a report may no longer be available to 
you, or even its existence recognized. Thus the steps summarized in Box 
5.1 and expanded below are useful in carrying out an effective structured 
skim reading. 


BOX 5.1 THE STRUCTURED SKIM READING: 7 STEPS 


Step 1 Register the complete reference. You may not gain access to 
the material again. 

Step 2 Study the title page. Note down the title, date of publication, 
institutions, etc. All this information helps to locate the political, so- 
cial and institutional context of the report. 


Step 3 The executive summary. This is good for reconstructing policy 
discourses and debates on the issues, but may not reflect the body of 
the report. 

Step 4 Skimming the substance. Differences between the body of the 
report and executive summary indicate areas of controversy and ne- 
gotiation. 

Step 5 Survey the quantitative data. For all useful data note the source, 
years, types of data and method of data aggregtion. Copy any useful 
primary data — it may be your only chance to do so. 

Step 6 Review the bibliography. This may provide useful references 
or give an indication of the depth of the research. 


Step 7 Note taking. Make notes about the report itself, about any 
useful information and of any points you think are important to verify 


or deepen in further reading. 
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Step 1 Register the complete reference 

The first look at possibly useful material should be done with pen (or pen- 
cil or word processor) at hand. For anything that is even remotely relevant 
to your study, note the complete reference. Much grey material is not cata- 
logued by library systems and has limited or restricted circulation. If you 
don’t record the complete reference now, there may never be another chance. 
Reports that seem useless in one phase of your research may become im- 
portant later on, and you have to know the titles and authors to find them 
again. Further, grey material often circulates in different drafts, with word- 
ing and even conclusions under debate. Note carefully the date of the par- 
ticular draft in hand. Even if you discard a report, it is useful to note the 
complete reference and to add a note as to why it is not useful, particularly 
if, like mine, your memory sometimes fails. When a title sounds particu- 
larly relevant to my study, I am more likely to forget that I have already 
chased down a reference and found it unhelpful. This can be time-consum- 
ing and embarrassing, especially if you must formally request authoriza- 
tion or have hounded a ministry official to see a particular report. 


Step 2 Study the title page 

The next step is a close look at the title page. Consultancy reports are not 
sold commercially, and so they don’t depend on catchy titles. The title is 
often a long and accurate guide to their content, as for example: State Farm 
Divestiture in Mozambique: property disputes and issues affecting new land 
access policy — the case of Chokwe, Gaza Province. Key words in titles 
sometimes indicate the institutional discourse in which they are rooted. 
Following the study by Cornia et al. (1987) Adjustment with a Human 
Face (which criticized some of the dimensions of structural adjustment 
programmes) for example, a rash of reports with ‘human face’ in the title 
appeared, generally concerned with alleviating poverty and minimizing 
the decline of social services under structural adjustment programmes. 


The title page often bears the names of the agencies who commissioned 
the research, and sometimes an indication of their relative importance to 
those who wrote the report. The report on state farm divestiture mentioned 
above, for example, was carried out by researchers from the University of 
Maputo, Land Tenure Centre for USAID, and the Government of the Re- 
public of Mozambique, Ministry of Agriculture, listed in that order on the 
title page. If, on further reading, the report proves interesting, you might 
want to investigate whether there were points of dissension between the 
two agencies, whether these were suppressed in the final report, and whether 
the viewpoint of the prime financing agency dominates the report. Reports 
are often parts of a series, so noted on the title page. In analyzing, for 
example, a recent poverty report on Lesotho, we need to read it in relation 
to the concerns and conclusions registered in other such similar reports 
undertaken through Social Dimensions of Adjustment (SDA) programmes 
in Africa. 
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The date of publication is often an important indicator of whether or not 
the report will be useful for a particular study. Mozambique’s structural 
adjustment programme was introduced at the beginning of 1987. A na- 
tional household income and expenditure study, based on surveys carried 
out in 1981, was published in 1984. Although this study may help us to 
understand change, we will not be able to use its data to establish rural/ 
urban income and expenditure gaps for a study of the consequences of 
structural adjustment for urban poverty. 


Step 3 The executive summary 

After inspecting the title page, move on to the body of the report. One of 
the particularities of grey material is that most consultancy reports pre- 
pared for international agencies have an executive summary, from 1000 
words to 10 000 words, on the main findings and arguments of the report. 
The prevalence of the executive summary is based on the realistic assump- 
tion that many of the policy makers for agencies that commission consul- 
tancy studies will not have the time, interest or technical background nec- 
essary for reading the entire report. The executive summary is thus a par- 
ticularly good source for reconstructing policy discourses and current de- 
bates on the issues we are researching. 


The executive summary may not be, however, a very good guide to the 
actual content of the report or a summary of its findings (see Chapter 3 for 
some reasons why this may have been done deliberately). Assumptions 
easily become conclusions when the evidence and argument to support 
them is compressed in a summary. An executive summary may conclude 
for example, that in the country under study poverty is principally a rural 
phenomenon. We cannot understand the meaning of this statement without 
reading the body of the report. It could mean only that most of the popu- 
lation live in rural areas and hence that most of the poor also live there, i.e. 
not a very interesting finding. Or it could mean that a higher proportion of 
people in rural areas fall below the poverty line than do those in urban 
areas. In this case we need to know how poverty lines were defined in rural 
and urban areas. Or, if we are dealing with a migrant labour system, the 
body of the report may show that rural incomes are dependent on remit- 
tances from urban areas and hence that the conclusion that poverty is prin- 
cipally a rural phenomenon is deceptive. Thus we use our background read- 
ing to note questions and issues raised by the executive summary that should 
be carefully examined in the body of the report. 


Step 4 Skimming the substance 

Given that the executive summary can be a polemical document with a 
message different from that of the body of the report, your skim reading 
should not stop with it. A quick look at the table of contents (or skimming 
through chapter sub-headings in the body of the report if there is no table 
of contents) may reveal that there is information not covered at all in the 
executive summary. Introductions and conclusions to chapters sometimes 
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phrase policy issues and summarize findings in language different from 
that of the executive summary. Differences may give us indications of ar- 
eas of controversy and negotiation. 


Step 5 Survey the quantitative data 

In your first structured skim of a report, check the list of tables, figures, 
and appendices, to see whether the report is likely to have quantitative data 
which will be useful in your study. If you see that there are data that are 
definitely useful, it is worthwhile to note the source, years, types of data 
and the way the data are aggregated. If the source is primary, you may not 
be able to find the data elsewhere. If the data come from a secondary pub- 
lished source, it is more reliable to obtain them from that source. If you 
have access to a photocopier, you can copy relevant tables, but often you 
must transcribe the data yourself, an arduous procedure. 


Some reports include lots of graphs, but not the source data. Note whether 
quantitative data are included in tables and appendices within the report. If 
there are only graphs, we can reproduce or interpret them in our own re- 
port, but we cannot verify the data nor analyse them in different ways. 
Authors of grey material use graphs because their readers can understand 
images quickly, but do not have the time or capacity to see patterns in 
quantitative data. Sometimes, however, motives are not didactic. The data 
may come from restricted sources and cannot be reproduced, or authors 
may know that in quality terms the data are not very reliable and should 
not be reproduced. And sometimes consultants withhold quantitative data 
because they consider them their intellectual property, to be used in future 
publications. 

Step 6 Review the bibliography 

Even in a quick skim of a report, it is useful to look at its bibliography. In 
our initial phases of research, it may provide references that we can follow 
up. Later on, when we ourselves know more about the topic, it may help us 
to determine whether the report was itself based on a good mastery of 
existing studies. If, for example, in reports on Benin or Mozambique, most 
references are in English, we can infer that the researchers did not speak 
French or Portuguese, the respective national languages, and thus that their 
understanding of local issues may be superficial. 


Step 7 Note taking 

After this first structured skim reading, you should know whether a report 
is useful and should be put aside (or photocopied) to be read again more 
carefully. At this point, stop to take three different kinds of notes. 


1 Notes about the report itself. Thus, in the example above, the fact that a 
poverty study on Mozambique sponsored by the World Bank observed 
that prospects for off-farm employment have never been very bright is 
an important fact in itself, whether or not we think that observation is 
correct. To determine whether the evidence of the report supports the 
conclusion, we have to dig into the report. 
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2 Scraps of relevant useful information that we have noted in passing, 
quite often quantitative data, in a report that will not merit further close 
study. A detailed study of a women’s sewing co-operative carried out 
for an NGO that sponsored the co-operative may, for example, contain 
data on overall membership in urban non-agricultural co-operatives that 
will be useful for a study on urban poverty. For both these first two 
types of notes exact citation and complete references to page numbers 
are important. 


3 Perhaps most important in interpreting institutional discourses, are the 
preliminary notes to ourselves about points we think are important to 
verify or deepen in further reading. These notes reflect what we bring 
with us to our reading — our theoretical baggage and what we have 
already learned about our topic and the area under study. 


5.5 LOCATING THE REPORT INSTITUTIONALLY 


To interpret institutional discourses, we have to locate a report institution- 
ally. This means placing it in its political and historical context, establish- 
ing the institutional identities of its authors and audience, and attending to 
the objectives and scope of the study as defined by its terms of reference 
and the ways in which research was carried out. 


Before beginning the substantive reading of the report, try to recreate the 
context in which the research was carried out and the report written. The 
date should serve as an indicator if we have already constructed a rough 
chronology of the recent history of the place we are studying in our minds, 
and reviewed recent policy debates around the topic under research. In 
development discourse around the position of women in African agricul- 
ture, for example, it was commonly assumed until the 1980s, following 
Boserup’s (1970) work, that women were subsistence producers. In the 
late 1980s, following work by Mackintosh (1989) and others, the impor- 
tance of women’s work in cash-cropping, even though men often control- 
led the cash, began to be recognized. If we are looking at a report on rural 
poverty in Africa, we can check to see whether the researchers simply as- 
sumed the subsistence role of women or actually looked at intra-household 
organization of both labour and income. 


Similarly, political issues can shape the language of the report, its silences 
as well as its focus. In Mozambique, for example, a gradual collectiviza- 
tion of agricultural production was official policy for many years. At the 
research centre, when we wrote on issues such as market access and the 
terms of trade for peasant producers, we argued that voluntary co- 
operatization would depend on the improvement in marketing position pro- 
vided for the broad mass of smallholders. Research reports written later 
under structural adjustment take the need to improve the marketing posi- 
tion of individual peasant producers as a given fact and devote much less 
attention to co-operatization. Gaps between the dates when the research 


118 


5 INTERPRETING INSTITUTIONAL DISCOURSES 


was carried out and when the report was published are often good indica- 
tors of political controversy and hence contested drafts. Locally based re- 
searchers often know about the existence of such controversies and are 
thus able to reconstruct the political context of a final report by locating 
omissions from it. 


If we have learned (from the title page or terms of reference) that the report 
is part of a particular series, we then think about where it fits and how that 
may affect its language and concerns. In the first round of poverty studies 
emerging from the critique of structural adjustment in Africa, for example, 
almost all reports took care to propose how well-defined and restricted 
groups should be targeted for special assistance. Hence there was a great 
deal of attention to the issue of whether or not it would be possible to 
identify and subsidize a staple (generally one in glut on world food mar- 
kets), which the poor and only the poor eat — such as sorghum or yellow 
maize. A recent World Bank report on structural adjustment in Africa has 
now concluded that poverty is so widespread that tightly targeted meas- 
ures probably will not satisfactorily reach the poor (World Bank, 1994). 
Interpreting what particular country studies say about targeting means lo- 
cating them within this debate. 


The questions and conclusions of a particular report reflect the political 
context within which they were written. Some of the most extensive work 
on poverty in Southern Africa was done in the series of Carnegie studies on 
poverty in South Africa co-ordinated by SALDRU, a research centre lo- 
cated at the University of Cape Town. In the early 1980s, these reports 
were written by researchers opposed to the apartheid regime, but their 
language reflects the context of political repression in which they were 
written and their recommendations show that few thought that by the end 
of the decade the regime would be politically ruined. Similarly, research 
on agrarian issues at the Centre for African Studies in the first decade of 
Mozambican independence was focused on the importance of understand- 
ing peasant production within an overall strategy of socializing produc- 
tion. In both cases the information contained in the reports continues to 
have relevance today, but some of their immediate concerns now seem 
anachronistic and limited. 


5.5.1 THE INSTITUTIONAL IDENTITY OF THE RESEARCHERS 

To interpret the report, we have to think about the institutional identity of 
the authors. Are they employees of the agency that commissioned the re- 
search or are they specialized researchers or academics contracted espe- 
cially for this research? Are they locally based researchers or were they 
part of an internationally recruited short-term consultancy team? If the au- 
thors are named, what do we know about their disciplinary background 
and their previous research experience? 


The name of Harold Alderman, for example, appears on several studies 
done on poverty and food-security related issues under structural adjust- 
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ment in Mozambique. He has done previous work for the World Bank and 
for the International Food Policy Research Institute (IFPRI) and was affili- 
ated with Cornell University when he did the studies in Mozambique. He 
did research on food issues in Pakistan and now works in various African 
countries. He is a neo-classical economist whose work is generally con- 
cerned with making markets work better. The solutions he recommends, 
such as the introduction of food-stamps, are marked by the institutional 
world of the United States; they would not be envisioned by Mozambican 
academics (unless trained in the United States), nor even perhaps by con- 
sultants from the IDS. His work tends to differ in its language from that of 
consultants from the Wisconsin Land Tenure Centre (LTC), another uni- 
versity-based applied research centre in the United States, who have also 
worked in Mozambique. LTC researchers are of course more specifically 
concerned with agriculture, they have varied disciplinary backgrounds in 
the social sciences, and they tend to be more critical of strategies of market 
liberalization than do those from Cornell. 


A close study of the bibliography can help us locate the institutional iden- 
tity of the authors — telling us something about both what sources were 
consulted and which sources were thought important to record. Current 
work undertaken in the framework of SDA more often refers to the work of 
anthropologists and sociologists than did previous World Bank studies — 
although arguments need not have changed. The critique by Ferguson (1990) 
of the 1975 World Bank Country Economic Report on Lesotho appears in 
the bibliography of a recent World Bank poverty assessment of Lesotho, 
even though its conclusions bear a much stronger resemblance to those of 
the 1975 report than to Ferguson’s. Since we approach grey literature al- 
ready armed with a good control on the field, we can also note significant 
gaps or silences in sources. Reports done by the Centre for African Studies 
on agrarian structure in Mozambique are rarely cited in consultancy stud- 
ies done under the aegis of the World Bank, in part because this work is 
associated with Frelimo’s discredited socialist period. 


5.5.2 IDENTIFY THOSE WHO PAID FOR THE STUDY AND THEIR INTERESTS 

Attending to how a study was funded helps to identify the audience the 
report was intended to reach and thus to interpret its findings and its si- 
lences. At one extreme are confidential reports commissioned by a specific 
agency that controls the choice of researcher and terms of reference of the 
study and has right of veto over acceptance of its results. In the research 
centre in Mozambique, by contrast, our salaries were paid by the univer- 
sity, and our research funded mainly by long-term grants from the Nordic 
countries. Although we responded to research requests from government 
agencies and always required authorization from both national and provin- 
cial authorities to carry out research, we did not have to submit drafts to 
government agencies before publication. This did not mean however that 
there was no self-censorship. We were careful to weigh what was said against 
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probable reactions; we did not want to be excluded from key national de- 
bates on agrarian strategy. We saw our audience, however, as much larger 
than a particular government agency; we tried (not always successfully) to 
write in a style that would be accessible to most educated Mozambicans 
and to produce reports that would introduce information about Mozam- 
bique into university teaching. 


5.5.3 INSTITUTIONAL CONSTRAINTS IN CARRYING OUT RESEARCH 

We can often tell a lot about the limitations of a study by analysing the 
terms of reference and what is said about when and how the research was 
carried out. There are usually references to the locations that the research 
team visited and a list of people who were interviewed. Even expressions 
of thanks may give us information about the scope and limits of the re- 
search. 


Sometimes the terms of reference of a study will show us that there is a 
link between a study and an ongoing project. This tends to affect the con- 
clusions of a report and sometimes even the questions it asks, particularly 
if those who have commissioned or execute the research are also responsi- 
ble for the execution of the project itself. Schubert's (1993) evaluation of a 
cash-transfer scheme for the alleviation of urban poverty in Mozambique, 
for example, has to be read with care since he was involved with the 
conceptualization and earlier evaluations of the scheme. 


Some actors have much more institutional presence than others in grey 
material, a fact which may already be clear from the terms of reference and 
sources of a study. Reports on international migration policy in Southern 
Africa, for example, tend to deal with the situation of documented work- 
ers, such as miners, whose movement back and forth from South Africa is 
registered. Undocumented migration, which for some countries in the re- 
gion probably exceeds legal migration, is notoriously difficult to research 
and hence is not discussed in reports. Solid quantitative data on formal 
migration can be obtained from the Chamber of Mines and from the gov- 
ernments of labour-supplying countries. Looking at the terms of reference 
and sources consulted in research will tell us quickly whether or not the 
report has attempted to grapple seriously with the issue of undocumented 
migrants. 


Locating where the research was carried out helps us to determine whether 
its findings are generalizable at a regional or national level. Reading a study 
on peasant marketing in a war-torn country like Mozambique with a good 
map in front of us, for example, we try to reconstruct the experiences of 
those rural communities studied during the war. Areas visited by consult- 
ants tended to be close to urban areas or to complexes with military protec- 
tion; they did not visit regions where they could not be protected from 
attack by Renamo, the armed opposition movement. 
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5.6 INTERPRETING INSTITUTIONAL DISCOURSES IN 
CONSTRUCTION OF ARGUMENT 

Grey material can be a privileged direct source of information about the 
ways in which policy issues are posed in our area of study — a good inform- 
ant — persuasive, sometimes deceptive, with fixed ideas not always revealed. 
Hence when we have identified grey material useful for our literature study 
and located it institutionally, we approach it in the same critically 
constructive way outlined by Barrientos in the previous chapter. Here, then, 
we will look at some specific issues in the interpretation of grey material 
in the construction of our own arguments. Box 5.2 summarizes problems 
in the interpretation of grey material that we have discussed in this chap- 
ter and what we do about each in the process of our literature study (steps 
1 to 5). 


BOX 5.2 OVERVIEW OF INTERPRETIVE TASKS 


Since grey material is Use of this material 

predictably limited because of: requires: 

* questionable but unstated * identifying and rethinking , 

underlying assumptions assumptions (step 1) 

* the questions it asks * formulating alternative 
questions and extending 
the range of possible 
answers (step 2) 

* illusive methodology * confronting the data: how well 
they answer old and 
new questions (step 3) 

© the conclusions it finds ¢ rethinking conclusions 

acceptable (step 4) 
* the audience it intends to * rethinking the audience 
reach (step 5) 


In the following sections, we will move through each of these steps in 
interpretation one by one, with examples of each. In reality, however, this 
process is never a linear set of sequential steps — it is an iterative process. 
Rethinking the answers helps us to locate and criticize implicit assump- 
tions; alternative interpretations of the data help us to locate methodologi- 
cal weaknesses in the analysis of the data, and so on. As your own argu- 
ment takes shape, what you find, and reject, in the report will change. 


Step 1 Identifying and rethinking assumptions 

Background reading in the history and political economy of the place we 
are studying helps us to uncover and rethink assumptions. We have dis- 
cussed how Ferguson’s (1990) previous reading on Lesotho, for example, 
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led him to challenge the assumption of the World Bank (1975) study that 
assumed that peasants of Lesotho were subsistence peasants weakly inte- 
grated in the market. 


Background reading on different theoretical approaches to our problem 
can help us uncover and recast assumptions. Some reports on rural pov- 
erty, for example, make general statements about what peasant households 
or smallholders do. There is an implicit assumption that the interests, atti- 
tudes and behaviour of all within the household are reducible to those of 
the male head of household. Feminist criticism of this assumption has led 
us to recognize that men and women often have different and contradictory 
interests within households. Recent studies of households in Africa in par- 
ticular emphasize that despite co-operation and male dominance within 
households there are domains controlled by women. 


Step 2 Formulating alternative questions and extending the range 
of possible answers 

Once we have recast underlying assumptions, we will also find alternative 
questions. Studies like that of Ferguson have led us to question the pre- 
sumption in studies of rural poverty in Southern Africa that rural problems 
have agricultural solutions. In other words, we no longer accept that if 
poor people live in rural areas, the way to improve their incomes is neces- 
sarily by improving their farming. Thus we may recast the questions in our 
study to focus on how to improve off-farm employment and non-agricul- 
tural activities in rural areas. Or if we refuse to assume that the household 
is a basic unit of analysis, we may choose to ask specifically how gender 
differences shape or express the experience of poverty. Posing new ques- 
tions means extending the range of possible answers or working hypoth- 
eses that we bring to interpretation of data. 


Step 3 Confronting the data: how well they answer old and new 
questions 

Once we have recast our questions, we can confront the data to see if they 
provide information on a new set of questions. Quite often, however, when 
we confront the data to see whether they allow us to answer the questions 
posed by the report and support its findings, we will also think further 
about assumptions and questions. This process is discussed by Mukherjee 
and Wuyts (Chapter 11). Here I will only signal some of the specific things 
to look for in confronting data presented in grey materials. 


First, attend carefully to the original source of the data and return to it if 
possible. 


Most World Bank studies (and many others) depend on the economic data 
collected and assessed by the country study team. In the country study you 
will usually find an explanation of how the data were collected, what the 
limitations were, which figures are based on estimates, etc. — information 
that tends to disappear in successive reproductions of data. 
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If you can locate the original source of data, you can check not only whether 
data have been correctly reproduced, but also if they have been selectively 
lifted. In interpretation of time series data, for example, the choice of a 
base year often establishes a direction of interpretation. A report critical of 
structural adjustment may find, for example, a sharp rise in infant mortal- 
ity in urban areas in the year after the introduction of a structural adjust- 
ment programme. But if we include the ten years prior to introduction of 
the structural adjustment programme, the data may show a long-term ten- 
dency toward rising infant mortality, associated, perhaps, with the break- 
down of public health services. 


Second, see whether the categories in which data are grouped correspond 
to the questions that are asked in the report and the answers found. Much 
grey material is not based on new surveys or independent research — data 
are found and made to speak. There is nothing wrong with this in itself 
(indeed, doing it is one of the prerequisites to finding out fast), but it means 
that we must be particularly careful about verifying interpretations. As the 
result of feminist criticism of research methods in the social sciences (cf. 
Eichler, 1988), for example, we are now much more careful about over- 
generalization and lack of gender and age sensitivity. Take for example, a 
study on the consequences of a cash-crop farming scheme for rural in- 
comes based only on interviews with household heads, most of them male. 
The study may tell us many things, but it cannot tell us about intra-house- 
hold distribution of income and thus it cannot generalize about the effects 
of the project on rural poverty. 


Third, integrate the qualitative studies you have read into the interpretation 
of quantitative data, and micro-level studies into the interpretation of macro- 
level studies. For example, where migration is clandestine, registers of 
migrants are unreliable sources of data on the extent of migration. Male/ 
female ratios can be an indirect indicator of emigration by men, as long as 
there is little emigration by women. To relate changes in male/female ra- 
tios to changes in migration patterns in Southern Africa, we therefore need 
qualitative information on the legal changes in the 1960s and 1970s that 
restricted access to women from Lesotho, Botswana and Swaziland to the 
South African labour market. Case studies and the collection of individual 
labour histories cannot be aggregated to provide a quantitatively secure 
picture of the extent or shape of migration, but they can alert us to patterns 
to probe at a macro-level and provide direct evidence on underlying mecha- 
nisms (see Chapter 14). 


Conversely, we can integrate quantitative data into the interpretation of 
qualitative data and macro-level information into the interpretation of mi- 
cro-level studies. In part because of its policy orientation, there is, often, a 
tendency towards over-generalization in grey material. Conclusions based 
on the phrases ‘most of or ‘the majority of cover a range from 50.1% to 
99.9%, and are not necessarily based on exclusive categories. Without quan- 
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titative data, it is impossible to determine the meaning of such phrases. A 
report suggesting that the majority of rural households obtain most of their 
income from agriculture may lead us to under-emphasize the importance 
of off-farm income for rural households that depend both on farming and 
non-agricultural work. Processes that may appear general and inevitable at 
a micro-level may become specific and transient when we locate them within 
a macro perspective. Sharply falling rural incomes in the 1970s in Mozam- 
bique did not neccessarily apply to communities in southern Mozambique 
relying heavily on remittances from migrant miners in South Africa whose 
real wages were rising. 


Step 4 Rethinking conclusions 

Having located assumptions, recast questions and possible answers, and 
confronted the data, you should compare the conclusions of a report with 
those you would draw yourself. There may be conclusions that are simply 
wrong or misleading, or there may be significant silences. The observation 
on off-farm employment in the report on poverty in Mozambique discussed 
above, for example, is not precisely false, if we think that the ‘brightness’ 
of employment prospects has to do with the levels and conditions of that 
employment. It is, however, deeply misleading if it is read to imply that 
off-farm employment is not central to the economy of many rural house- 
holds. 


In interpreting policy-oriented documents your conclusions about the find- 
ings of the document become not only a basis for developing your own 
arguments, but information about the world you are studying. In the case 
above, for example, the fact that an important World Bank report on Mo- 
zambique generalized about all of sub-Saharan Africa and emphasized the 
marginality of off-farm employment is itself a finding about an important 
institutional actor. 


Step 5 Rethinking the audience 

Sometimes it is easy to identify the audience for which a report was written 
and to recast its information for a different group. An evaluation report on 
a particular development project, for example, will attend to the compe- 
tence of particular project staff. If we are writing a more general assess- 
ment on the co-ordination of NGO activity in rural poverty alleviation for 
the Ministry of Planning, we are not interested in the details of individual 
competence, but in the recruitment process that underlies general prob- 
lems in staff competence in NGO projects. We may draw important eco- 
nomic information from a highly technical report on public sector manage- 
ment written for economists at the World Bank and present it in simpler 
and more accessible ways for public sector trade unions. 


Sometimes, however, rethinking the audience means not only identifying 
the objectives and experience of different actors in the policy process, but 
also recognizing that institutional discourse itself may have changed. 
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A Mozambican researcher assigned to work with an early World Bank mis- 
sion on agriculture told me, for example, “They think that all Mozambican 
peasants are subsistence cultivators, so I’m going to take them to Manica 
to see specialized commercial growers’. Later, when the stereotype of the 
entrepreneurial smallholder had replaced that of the traditional tribesman 
in the minds of international consultants, it became important to take them 
to see women-headed households dependent on wages and remittances. 
Reports that seem anachronistic or limited in their concerns can become 
relevant when we locate their findings within a different set of issues for an 
audience with different expectations and experience. 


5.7 CONCLUSION 


This chapter has emphasized that grey materials — policy-oriented reports 
prepared for and by governmental and non-governmental agencies — can 
be important sources of information. We must treat them as we would other 
key informants — locating them institutionally, reconstructing the context 
in which they spoke, analysing and interpreting both their statements and 
their silences. This is only possible if we are both informed and self-con- 
scious about our own assumptions and concerns in the developing dia- 
logue that we pursue with all our sources — and our own projected audience 
—in the construction of policy-oriented arguments. 
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THINKING WITH PEOPLE AND 
ORGANIZATIONS 


6 PEOPLE AS INFORMANTS 


PHILIP WOODHOUSE 


This chapter is concerned with investigative approaches which develop- 

ment managers and policy investigators need when ‘finding out’ from other 

people. In it I shall consider two basic issues: 

* Why should development managers and policy investigators be particu- 
larly interested in using people as a source of information? 

* What methods can be used to improve the accuracy and reliability of 
ideas and information people provide? 


I wish to emphasize at the outset that I consider these issues from the stand- 
point of the investigative development manager; that is, from the perspec- 
tive of the initiator of the investigation — the one who wants to find out. 
Thus, my concern with accuracy is: does the information people provide 
tell me what I want to find out? Similarly, my concern with reliability is: 
can I have confidence that what I have found out will not be contradicted 
or easily undermined by other, similar investigations? My prime concern 
is, therefore, with the rigour of investigation where people are the source 
of information. 


It is important to note that this leaves largely for the next chapter a key 
issue in using people as information sources, that of power. Power re- 
lationships are incorporated in all forms of knowledge, even those in docu- 
mentary form, as discussed in the previous chapter by Bridget O’Laughlin. 
However, working with people brings out power relations in a very imme- 
diate way. Investigators determine what information is being sought and 
what it will be used for. Informants decide how much information to dis- 
close. Seeking information from people is always, therefore, an interactive 
process, and the investigator stands to find out more from it if it is also a 
collaborative process. If informants are to collaborate, should their partici- 
pation in the investigation extend to shaping the purpose of the investiga- 
tion, and to deciding what information is considered valid? Some have 
argued (Chambers, 1994a) that when working with disadvantaged groups 
(for example, with the rural poor in Africa or India) investigation should 
always seek to ‘empower’ those providing the information. That is, devel- 
opment managers’ investigation should encourage informants also to ‘find 
out’ by prompting them to reflect and analyse their own experience. From 
this ‘participatory’ perspective, development managers should be ‘conven- 
ors, catalysts, and facilitators’ (Chambers, 1992, p. 12). 
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I shall return later to the issues of rigour raised by participative enquiry 
and other techniques used in finding out from people. I wish to emphasize 
at this point that for the purposes of this chapter I shall step back from any 
such assumptions about the objectives of development managers’ investi- 
gative activity, and simply recognize that these will be determined by the 
investigator or by those to whom she or he is accountable. There are clearly 
a number of factors which will operate in shaping these objectives, includ- 
ing the legislative and policy framework, and contractual obligations, within 
which the development manager works, and the political values which she 
or he holds — for example, in relation to environmental conservation or the 
living standards of disadvantaged social groups. One consequence of step- 
ping back from making assumptions about the purpose of investigation is 
recognition that the investigator's purpose is not necessarily shared by all 
those from whom information is to be sought. Thus, while not excluding 
the possibility that a development manager's goal is to empower people 
who act as informants, my discussion in this chapter does not assume that 
this is always the case. 


The chapter is organized in the following way. Section 6.1 examines why 
development managers should particularly want to turn to people as sources 
of information. Section 6.2 looks at ways of dealing with difference and diver- 
sity in the information people provide, contrasting the two major approaches 
of structured survey and semi-structured interview. Section 6.3 discusses rig- 
our with respect to these two contrasting approaches. Section 6.4 explores 
ways of improving communication between investigators and informants. 
Section 6.5 considers issues of accuracy and reliability raised by participatory 
enquiry, with particular attention to participatory rural appraisal (PRA). 


6.1 WHY FIND OUT FROM PEOPLE? 

It may seem obvious that development managers need to consult people 
whom their decisions affect, but it is worth reflecting briefly that people 
know things that development managers would be unable to find out from 
any other source. Two types of information of this kind can be identified. 


Firstly, development managers may wish to seek knowledge not available 
elsewhere. Examples of this might be oral histories of rural communities, 
details of the rules governing customary land tenure, or indigenous techni- 
cal knowledge relating to the use of plant and animal species. 


Secondly, development managers may need to know the perceptions of indi- 
viduals or social groups in their capacity as users of services or resources, and 
how these are manifest in particular patterns of decision-making. 


Some issues that development managers may need to investigate may be a 
mixture of both. For example, farmers’ views on agricultural topics may be 
sought both because farmers have technical insights into the behaviour of plants, 
animals and soils, and also because farmers’ perceptions of their priorities 
may dominate the way these resources are used. Or the staff of a development 
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agency may know the inside story of how the agency works and also have 
preferences with respect to its various stated development aims. 


It is worth noting here how this emphasis differs slightly from that of Joseph 
Hanlon’s account in Chapter 3 of a journalist’s approach to ‘finding out’. 
Both use people as sources of information but whereas Hanlon’s journalist 
seeks ‘those who know’, my concern is also with the knowledge and per- 
ceptions of all those who have some interest in, or are affected by, the issue 
being investigated. In many cases this means finding out about the people 
concerned themselves, as a group. In other words, I am concerned with 
ways of finding out not only with those who fund, plan, design and imple- 
ment an irrigation scheme, say, but also with all whose livelihoods are 
affected by the scheme. I identify ‘who is important’ to the enquiry not only 
by ‘what they know’ but also by ‘who and where they are’ — that is, their 
position in a particular social situation. Having made this distinction, I must 
hasten to add that I am only distinguishing between the emphases in the 
two chapters. I am not making a distinction between journalists and other 
types of investigators: each will seek informants according to the purpose 
of the investigation. The approach taken in this chapter should reinforce 
the point that, for development managers, understanding the perceptions of 
‘non-experts’ may be as important as understanding those of ‘experts’. 


Setting out in these terms the need to find out through talking to people 
reminds us of the ‘subjectivity’ of information which people provide. It can 
be argued that all knowledge is subjective. That is, what we know is shaped 
by how we know it, as illustrated by Blackmore and Ison’s images of a 
‘grassland system’ through the eyes of agronomist, ecologist, sociologist 
and animal scientist in Chapter 2. This subjectivity is even more strongly 
apparent when knowledge is generated from an encounter between ‘inves- 
tigator’ and ‘respondent’. An agronomist asking a farmer about farming 
problems, for example, will obtain an understanding which is the product 
not only of two (possibly quite different) perceptions of farming activity, 
but also of the efficiency of communication between them. Here, again as 
noted by Blackmore and Ison, ‘message sent’ is not the same as ‘message 
received’. Section 6.4 explores techniques for improving the efficiency of 
communication between investigators and their informants. 


For those investigating development issues by using ‘people as inform- 
ants’, an important consequence of the subjective nature of information is 
that informants’ views will tend to be diverse and possibly contradictory, 
reflecting differences in what individuals know and also how that knowl- 
edge affects their interests. During the course of an enquiry, development 
managers are likely to accumulate a large amount of information, much of 
it from quite divergent standpoints. A key question in investigative ap- 
proaches which involve seeking information from people is, therefore, that 
of how to manage the diversity of views which will be generated: how to 
reduce the data to an amount that can be understood. 
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6.2 DEALING WITH DIVERSITY: STRUCTURED SURVEY AND 
SEMI-STRUCTURED INTERVIEW 

This section will look at strategies for dealing with the diversity of re- 
sponse that inevitably confronts an enquiry seeking people’s views. It is 
organized under four main headings: 

* defining the population; 

* deciding who to speak to; 

* formulating questions to ask; 

* analysing the response. 


6.2.1 DEFINING THE POPULATION 

By ‘defining the population’ I mean identifying the boundary from within 
which the investigators will select people to take part in their enquiry. Where 
this boundary is drawn will depend primarily on the purpose of the en- 
quiry. As I observed above, this may involve decisions about the relative 
importance of ‘expert’ and ‘lay’ views. Is the investigation concerned, for 
example, with the whole range of interests and opinions in a given geo- 
graphical area, or is its purpose to investigate the views of a particular 
social group (e.g. women, landless households, charcoal-makers), perhaps 
one whose interests are felt to have been underrepresented in decision- 
making? Is it sufficient, like Hanlon’s journalist, to find out second-hand 
from ‘those who know’, or is it necessary to find out directly about a par- 
ticular group or population? 


Drawing this boundary may require some preliminary investigation in or- 
der to find out who should be involved in the enquiry. In a recent study in 
Mali (Chevenix-Trench, Tessougué and Woodhouse, 1997), for example, 
the objective was to interview a sample of rice cultivators in order to un- 
derstand the impact of rice-growing — a new activity in the area — on their 
livelihoods and on land tenure. Initially, it was assumed that all the cultiva- 
tors would be living locally. However, preliminary enquiry revealed that 
some cultivators were seasonal migrants from distant villages. The bound- 
ary of the enquiry therefore needed to be extended from the immediate 
vicinity of the rice fields to include these distant villages. 


6.2.2 DECIDING WHO TO SPEAK TO 

Investigators select their informants according to the type of information 
they are seeking, which, once again, is determined by the purpose of the 
enquiry. There are two principal, quite distinct, approaches which may be 
followed. The first of these is that of a sample survey, the second is that of 
focused interviews. The decision as to which approach to use depends on 
what kind of information is sought. It is quite possible that both approaches 
could be useful in a single enquiry, but it is important to be aware of the 
strengths, weaknesses, and very different criteria for establishing rigour, in 
each approach. The main characteristics of the two approaches are set out 
later in Table 6.2. 
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The point of a sample survey is to generate a description of a wider popu- 
lation without actually talking to every individual in the population. It is 
important to note that the objective here is mainly descriptive, providing 
answers to ‘What?’ questions (e.g. number of households owning livestock 
or cultivating more than one hectare of land, number of households with 
off-farm income, percentage of a workforce satisfied with their jobs, etc.), 
and that the emphasis is generally on quantitative aspects, although some 
qualitative data can also be obtained. Sample surveys can be useful in con- 
texts where information is needed on populations with large numbers of 
people, or where comparative data are needed. 


In order to describe the wider population, the sample of informants for a 
structured survey is selected, on probability criteria, to be representative of 
the population (see Box 6.1). Probability sampling requires a ‘sampling 
frame’: that is, a means of identifying all the elements of the population 
so that they can be included in the sampling procedure. In the simplest 


BOX 6.1 PROBABILITY SAMPLING 


Atits simplest, probability sampling means the sample of informants 
is selected at random in the population. In practice, populations usu- 
ally are known to contain subgroups which are significant to the pur- 
pose of the survey, and these subgroups are used to ‘stratify’ the sam- 
ple: that is, the informants are selected from subgroups, either fol- 
lowing the proportions which each subgroup represents in the popu- 
lation as a whole or deliberately favouring certain subgroups known 
to be more important to the study than their numbers would indicate. 


For example, in the Mali study mentioned above, the sample of rice 
cultivators in a single village was stratified among five subgroups. The 
sample aimed to include 20% of cultivators in each category (Table 6.1). 


TABLE 6.1 EXAMPLE O! 
INA 


RATIFIED SAMPLING FOR RICE CULTIVATORS 
GLE MALIAN VILLAGE 


Cultivator subgroup Number Number in sample 
in village (% of subgroup total) 
1 Heads of extended households (managing 
fields for household consumption) 88 19 (22%) 
2 Individual men (married and 
unmarried) cultivating on their own 


(or immediate family’s) account 65 13 (20%) 
3. Individual (married) women 119 24 (20%) 
4 Locally resident immigrants 55 10 (18%) 
5 Seasonal immigrants 80 17 (21%) 


Data from Chevenix-Trench, Tessougué and Woodhouse (1997). 
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version, they all have an equal probability of being included in the sample. 
While a sampling frame may be obtainable from existing documentary 
sources, such as census data or an organization’s employment records, these 
need to be checked for inaccuracies and omissions. It is quite possible that 
in remote and rural areas a sampling frame can only be obtained by under- 
taking from scratch a form of census of the population to be sampled. This 
was, in fact, the case in the Mali study of rice cultivators. 


In the other main approach — focused interviews — selection of informants 
is governed by the need to identify as wide a range of different viewpoints 
as possible, or by Hanlon’s criterion of talking to ‘those who know’, but 
the selection is not necessarily linked to the wider population in any quan- 
titative sense. The numbers of informants from minority social groups, 
whether powerful élites or socially marginalized, may be disproportionate 
to their presence in society as a whole. Thus, in a rural community the 
same number of informants may be sought among the 80% of households 
owning land as among the 20% without land. As a result, the number of 
informants can be much lower than for a sample survey. 


There are two main variants to the approach. One is referred to as purposive 
or theoretical sampling, where different subgroups are sampled more or 
less heavily, depending on their importance to the purpose of the research 
or to certain theoretical distinctions which the investigator is testing (e.g. 
in the above example, how strategies for coping with drought differ be- 
tween the landless and those with land). The other is the use of key inform- 
ants (‘those who know’) who are not necessarily representative of a popu- 
lation in any sense, but are chosen simply for their knowledge or distinc- 
tive viewpoint. In both variants, selection of informants is driven by the 
investigators’ need to test or clarify elements of their evolving hypotheses 
about the issues they are investigating. 


This idea of evolving hypotheses highlights a further distinction between 
sample survey and focused interview approaches. Whereas sample survey 
requires the sample of informants and the questions to be asked to be de- 
fined beforehand, the focused interview approach allows investigators to 
decide who to talk to, and what to ask them, as their understanding is shaped 
by earlier interviews. 


6.2.3 FORMULATING QUESTIONS TO ASK 

As with deciding who to speak to, there are two main approaches to formu- 
lating what to ask: the structured questionnaire and the semi-structured 
interview schedule. 


Structured questionnaires are generally used in sample surveys, primarily 
to answer ‘What?’ questions. The questions are standardized. Mostly of 
the ‘tick the box’ variety, they may be phrased ‘Which of the following...?’, 
‘How much...?’, ‘To what extent...?’, and so on, where the boxes to be 
ticked give a choice of options or a number of points on a scale. Question- 
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naire design is a detailed technique in its own right, requiring various types 
of expertise. Individual questions may need testing, and the overall design 
and mode of ‘administration’ (face-to-face, self-administered, postal, tel- 
ephone) has to be adjusted to the circumstances so as to obtain a high per- 
centage and reliable response. 


Can structured questionnaires also be used to answer ‘Why?’ or ‘How?’ ques- 
tions? One possibility is to analyse the relations between the answers to differ- 
ent ‘What?’ questions (see “Analysing the response’ below), although as we 
will see it is difficult to be sure about causal relationships by this means alone. 
Another possibility is for the questionnaire to include such questions directly: 
e.g. How did you negotiate to borrow land? Why do you lend land to people 
from outside your village? How do present farming practices represent a change 
in relation to the past? Questions such as these can be answered by means of 
predefined options or in the respondents’ own words, in ‘open ended’ format 
questions. Quantification is less straightforward, however, if the latter is cho- 
sen. In practice, answers to such questions tend to be complex, requiring fur- 
ther questions for clarification, and, unless investigators already know their 
informants fairly well, it can be difficult to frame such questions in a manner 
that is both appropriate and standardized. 


For these reasons, structured questionnaires are often an ineffective way of 
answering ‘Why?’ or ‘How?’ questions, which are better addressed using a 
semi-structured interview schedule. This allows investigators to explore 
issues with informants in a much more flexible way, using supplementary 
questions to clarify complex responses, and developing new lines of en- 
quiry — perhaps unforeseen but relevant to the investigator’s purpose — as 
these arise during interviews. The semi-structured interview approach is 
also useful for answering ‘What?’ questions where investigators are unfa- 
miliar with the social situation in which they are working. By providing 
answers to ‘Why?’ and ‘How?’ questions, semi-structured interviews al- 
low investigators to go beyond describing ‘what is’ to develop theories or 
hypotheses about the nature of relationships and decision-making which 
explain why people think or act they way they do. Addressing ‘Why?’ and 
‘How?’ questions leads to an emphasis on qualitative information in semi- 
structured interview approaches, although not necessarily excluding quan- 
titative data (as exemplified by PRA in Section 6.5). 


A semi-structured interview schedule generally has a number of headings 
or standard but open-ended questions, but each interview will vary in the 
way the investigator follows up interesting lines of enquiry that could not 
have been foreseen, using supplementary questions. It is possible to use 
semi-structured questioning within a sample survey, perhaps as an adjunct 
to a structured questionnaire. However, it is more usual for this form of 
questioning to be part of the technique of focused interviewing, where, as 
described above, those interviewed are either a purposive sample or else 
key informants, who may be chosen to throw light on new issues brought 
up by previous informants. 
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Thus the most usual approaches to finding out from people are, on the one 
hand, sample surveys done by administering structured questionnaires 
(‘structured survey’ for short), and, on the other, focused interviews using 
semi-structured interview schedules (‘semi-structured interview’). In the 
latter case, the ‘design’ of the approach, in terms both of adding new inter- 
viewees and of asking new questions, continues while the enquiry is in 
progress. This ‘iterative, learning-process’ design provides great flexibil- 
ity, and makes the semi-structured approach particularly suited to situa- 
tions where development managers are trying to understand unfamiliar 
social situations. It has seen wide application in rapid rural appraisal and 
participatory rural appraisal. 

6.2.4 ANALYSING THE RESPONSE 


In any investigation a major task is to gather together the ideas and the 
evidence that the enquiry has generated in order to summarize its findings. 
In the case of structured surveys, each informant is asked an identical set 
of questions, and the responses are recorded on specially prepared ques- 
tionnaire forms. The questionnaire responses are collated and subjected to 
statistical analysis, usually using a computer. The response to any given 
question can then be expressed as a statistic (mean and standard error) for 
the sample or subsets of it, taken to be representative of the population. 
Characteristics of the population are ‘inferred’ from statistical analysis of 
the sample. Statistical analysis will also show the extent to which certain 
characteristics of the population are associated, or correlated, with other 
characteristics. For example, in the study of rice cultivators in Mali, statis- 
tics showed that mean rice production per hectare was lower on hoe-culti- 
vated plots than on those tilled by ox-drawn plough. Statistical analysis 
also provides clear indications as to how significant such differences are: 
that is, how much confidence we can have that they are not the result of 
some chance pattern occurring in the response data. 


Note that above I stated that statistical analysis shows correlations. It does 
not prove causality. However, this is where attempts may be made to throw 
light on certain ‘Why?’ (or ‘How?’) questions through relating the answers 
to different ‘What?’ questions to each other, as signalled above. Although 
statistical correlations may be useful in exploring such questions, this type 
of analysis will not give definite answers by itself. Thus, though higher 
rice yields were correlated with cultivation by ox-drawn plough, this does 
not prove that ploughing with oxen was the cause of higher yields (even 
though this may be a reasonable supposition). It is equally possible that 
those cultivators wealthy enough to plough with oxen had other advan- 
tages, such as better quality seed, more labour for weeding, or plots with 
better access to water, which were a more direct cause of higher rice yields. 
Ata further extreme of interpretation, cultivators may have become wealthy 
enough to plough with oxen because they fortuitously occupied more pro- 
ductive plots of land. This indicates that great care should be exercised 
when using responses to ‘What?’ questions to answer ‘Why?’ questions. 
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This brings us back to weaknesses in structured surveys. While they can 
provide rigorously defined descriptions, they are notoriously open to mis- 
interpretation, giving rise to the phrase ‘lies, damn lies, and statistics’. A 
crucial point to bear in mind is that the interpretation of statistics from 
structured surveys is only as good as the qualitative information available 
to guide interpretation. An example from the Mali study (Example 6.1) 
demonstrates that the influence of households’ wealth and status on the 
productivity of their rice-growing only became apparent when statistical 
comparison was guided by information from semi-structured interviews, 
in this case villagers’ own criteria of who was wealthy and who was poor. 


EXAMPLE 6.1 INTERPRETING SURVEY DATA USING 
INFORMATION FROM SEMI-STRUCTURED INTERVIEWS 


At the outset of the structured survey of rice cultivators, it was 
believed that inherited access to land suitable for rice cultivation was 
a key indicator of wealth and status. However, a comparison of pro- 
ductivity (in terms of mean weight of rice produced per hectare) on 
inherited and borrowed plots showed hardly any difference (1.2 t/ha 
on inherited fields, against 1.4 t/ha on borrowed fields), suggesting 
that the wealthy were no more productive than their poorer neigh- 
bours. However, in the course of semi-structured interviews, village 
elders had been asked to rank all households according to wealth. 
They had identified three categories of wealth in the village: 


1 Wealthiest: self-sufficient in food, and owning more than five head 
of cattle. 


2 Middle: self-sufficient in food with one or two head of cattle. 
3 Poorest: food-deficit households. 


Considering each household in turn, they had allocated 51% to the 
first category, 32% to the second, and 17% to the third. Since it cov- 
ered all households in the village, the wealth ranking of each house- 
hold included in the structured survey sample could be used in statis- 
tical analysis. When this was used to compare rice productivity, the 
mean figure for the category of wealthiest households was more than 
double that of the category of poorest households, with that for the 
middle category lying in between (1.2 t/ha, 1.0 t/ha and 0.5 t/ha). 


Example 6.1 illustrates the important point that in many cases the descrip- 
tive data from structured surveys can only be accurately interpreted — and 
hence give valid findings — for populations with which investigators are 
already fairly familiar. Where this is not the case, a structured survey can 
be preceded by semi-structured interviews in order to establish an adequate 
‘theory’ with which to design the structured survey and interpret the statis- 
tics it generates. 
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6.3 RIGOUR IN STRUCTURED SURVEY AND SEMI-STRUCTURED 
INTERVIEW 

Structured surveys are often criticized for giving invalid results, as with 
certain well-known examples of opinion polls of voting intentions in the 
UK giving incorrect predictions of election results. This is similar to the 
mistake of assuming correlation implies causality. There are two aspects to 
establishing the reliability of results supposedly obtained from structured 
surveys: how the survey results are used to obtain a description of the popu- 
lation, and an interpretation of that description. 


Thus, in the example of electoral opinion polling, a properly conducted 
sample survey can tell us reliably what proportion of the population say 
they will vote for a certain party. Whether that can be interpreted to mean 
they will in fact vote that way depends on whether some of them were 
misinterpreting the questions (e.g. answering “Which party do you prefer?’ 
rather than ‘Which party will you vote for?’, which might be different), 
giving misleading answers, or answering in a way they thought would please 
the interviewer (‘interviewer bias’). 


Rigour in interpretation requires a structured survey to be backed up with 
other, perhaps more qualitative, information, that might be obtained as just 
suggested by combining the survey with semi-structured interviewing. 
However, rigour in description is much more straightforward, particularly 
if interviewer bias is minimized through careful wording and paying atten- 
tion to how the interviewer is perceived by the interviewee (see Section 
6.4 below). One of the undoubted attractions of the structured survey ap- 
proach is that, however potentially misleading — or invalid — the interpreta- 
tions, the procedures for undertaking and presenting the investigation are 
subject to definite rules. This means that, so long as rules of probability 
sampling are followed, the statistics produced by the survey can be taken 
as a reliable description of those aspects in the population as a whole. Thus 
in the rice cultivator study, it could be said with confidence that hoe-culti- 
vated plots had less rice productivity, even though one could not say why. 


Conversely, one of the consequences of the flexibility of semi-structured 

interview approaches is that procedures for establishing rigour are less sim- 

ple. In the ethnography literature from which semi-structured approaches 

were first developed, considerable attention has been devoted to discuss- 

ing ways of improving rigour (Robson, 1993). Essentially, the investigator 

needs to ask a number of questions: 

¢ Are my findings compatible with all the evidence? 

¢ To what extent could other investigators reproduce and corroborate my 
findings? 

¢ How far can my findings be applied to other situations? 


For investigative development managers to be able to answer these ques- 
tions, two main principles need to be borne in mind: triangulation and 
documentation. 
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Triangulation refers to the use of multiple sources of information to test 
and modify one’s understanding (or ‘theory’) of a given problem or situa- 
tion. In practice this involves using different methods of enquiry, different 
informants, and different investigators to see whether the ideas and infor- 
mation they generate can be accounted for by the developing theory of the 
issue under investigation. In semi-structured interview approaches, this 
forms part of an iterative cycle of interview, analysis, evaluation, and de- 
sign of the next interview. Each interview is thus used to test the ‘working 
hypotheses’ which have originated from earlier interviews. This process 
should be pursued until the working hypothesis can credibly account for 
all the available ideas and information, and further interviews confirm, 
rather than modify, this hypothesis. The rich repertoire of interviewing and 
other techniques that has developed in RRA and PRA, which are reviewed 
below, provides additional scope for triangulation by providing alternative 
ways of visualizing and interrogating social activity. Although some of 
these techniques involve group rather than individual responses, they can 
be considered within the broad framework of the semi-structured inter- 
view approach. However, the principle of triangulation should also remind 
investigators pursuing their enquiry by talking to people that they need to 
cross-check their interview findings against other relevant sources, such as 
earlier research findings, documentary material, and so on. 


Documentation refers to the need to record not only the substance of what 
informants say, but also key features of the investigation ‘process’, such as 
interpretation of what informants say which incorporates investigators’ 
observations on the context of interviews, and reflections on investigators’ 
(changing) rationale and understanding of the enquiry. The basic concept 
here is that ideas and information can only be understood with reference to 
the circumstances in which they were generated. This is the same as the 
notion that ‘what we know is shaped by how we know it’ mentioned earlier 
in Section 6.1. It means that if you wish to assess how far your findings 
may be applicable elsewhere, or reproducible by other investigations, you 
need to be able to examine the possibility of differences not only of subject 
matter and methodology, but also in the enquiry process itself. In order to 
do this effectively, investigators need to reflect upon their own role and 
influence on the interview, and the nature of the communication between 
themselves and those whom they are interviewing. I return to this need for 
‘reflexivity’ in the next section. 


In this section and the previous one I have considered investigation which 
relies on people as informants in terms of two principal approaches: struc- 
tured survey and semi-structured interview. Table 6.2 summarizes the dif- 
ferences between the two approaches. I have organized the discussion this 
way because the two approaches differ fundamentally in the way rigour is 
established. Structured survey uses theories of probability to select a sam- 
ple of informants whose circumstances and perceptions are taken to be 
representative of a wider population. Valid structured surveys require con- 
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siderable prior knowledge of the population involved in the enquiry, how- 
ever, both to select a sample and to interpret the survey findings. Semi- 
structured interview approaches select informants in order to obtain a wide 
range of viewpoints. Validity is sought not through statistical analysis but 
by generating explanations (hypotheses) to account for what informants 
have said and pursuing further information until investigators feel able sat- 
isfactorily to explain all the evidence the enquiry has yielded. The reliabil- 
ity of the findings of semi-structured interview approaches is further estab- 
lished through careful recording of the context and process of the enquiry. 


It needs to be recognized that discussion of the reasons for choosing one or 
the other approach is frequently polarized. The view advanced here is that 
one approach is not better than the other, but that they produce different 
kinds of information and thus are suited to different types of investiga- 
tions. To undertake either method rigorously needs considerable attention 
and an awareness of the method’s strengths and weaknesses. In practice 
the two approaches are often used to support one another. A common pro- 
cedure has been illustrated already in the Mali study where semi-struc- 
tured interviews with villagers identified what kind of people were rice 
cultivators, the extent of the village’s rice cultivation lands and the forms 
of land tenure. This constituted the basis for designing a structured survey 
of the way rice was grown on those lands. Equally, however, semi-struc- 
tured interviews can be used to follow up and explore questions suggested 
by gaps or contradictions in the findings of a structured survey. In the Mali 
study the structured survey suggested that the practice of charging rent to 
those borrowing rice land was one such question that needed further ex- 
ploration through semi-structured interviews. 


In these instances semi-structured and survey approaches are used 
sequentially. However, as mentioned above, elements of semi-structured 
interviewing may also be undertaken concurrently with structured surveys, 
by including ‘open-ended questions’, in which the form of response is 
largely determined by the informant, as part of a questionnaire schedule. 
Alternatively, separate semi-structured interviews may be conducted with 
some of the informants taking part in a structured survey. In practice, there- 
fore, the clear separation between structured survey and semi-structured 
interviews may become blurred. Even where combined in a single enquiry, 
however, it is essential that investigators retain a clear distinction between 
which information is being generated by which approach, for this will de- 
termine the process through which findings need to be validated. 


Awareness of the differences and scope for complementarity between 
approaches is particularly important when evaluating claims that semi-struc- 
tured interview approaches are more rapid or more cost-effective than struc- 
tured survey. Firstly, the methods produce different kinds of information 
and one should not be seen as substituting for the other. Secondly, semi- 
structured interviews require a much higher level of decision-making dur- 
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TABLE 6.2 STRUCTURE EY AND SEMI-STRUCTURED INTERVIEWS COMPARED 

Structured survey Semi-structured interviews 

Design completed before Design continues in interviewing phase 

interviewing starts 

All informants selected before Informants identified progressively, 

interviewing begins making use of findings from earlier 
interviews 

All questions defined before Questions are identified and modified 

interviewing begins in response to understanding or 
information gained in earlier 


interviews. New questions or aspects of 
enquiry may emerge during the course 
of an interview 


Identical question set Questions or topics of interviews 
(questionnaire) addressed to all tailored to different informants, and 
informants different stages of the enquiry (e.g. to 
elaborate or check on earlier findings) 
Informants chosen as Informants chosen to explore a range of 
representative sample of the different types of knowledge or 
wider population following perspectives, not linked to a 
probability criteria quantitative representation of the wider 
population 


Data ‘reduced’ to understandable —_ Data ‘reduced’ to understandable 
patterns using statistical analysis _ patterns by comparing what different 
informants say about specific themes or 


questions 
Findings validated by probability __ Findings validated by triangulation: 
criteria: the calculation of the does information from different sources 


likelihood that patterns observed _lead to the same conclusions? 

in the data have arisen by chance 

Most suited to answering ‘What?’ —_ Well suited to answering “Why?” 

questions, less effective in questions. Can answer ‘What?’ 

answering ‘Why?’ questions questions but findings cannot be used to 
calculate (infer) the distribution of 
characteristics in the wider population 


ing the course of each interview than does the questionnaire in the struc- 
tured survey. Consequently, the level of training, experience, and maturity 
required of field interviewers is much higher in semi-structured interviews 
than in structured surveys. If this is not recognized, and there is evidence 
that in many cases it is not (Cornwall and Fleming, 1995), the invalid 
structured surveys widely criticized in the past will merely be replaced by 
unreliable semi-structured interviews in the future. 
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6.4 IMPROVING COMMUNICATIONS 


I suggested at the start of this chapter that all investigations which use 
people as sources of information are necessarily interactive, and that they 
are more likely to be successful if they are collaborative. Investigators there- 
fore need to pay attention to establishing a collaborative context for the 
interview. A key aspect of this is informants’ perceptions of an investiga- 
tor’s identity and the purpose of the enquiry. This may not be as straightfor- 
ward as it might first appear. Hanlon’s remarks about personal goals and 
agendas are relevant here, and, as he observes, decision-making is easier 
when agendas are made explicit. However, I noted at the beginning of this 
chapter the possibility that a development manager may need to ‘find out’ 
from people who may be unsympathetic to that manager’s agenda. In such 
cases the investigative manager may prefer to identify herself or himself 
with a different agenda, more likely to be viewed positively by potential 
informants. This raises ethical questions to which I shall return, but a more 
immediate concern is that whatever identity an investigator adopts she or 
he needs to be clear about the purpose of the enquiry. I am not asserting 
that the goal of the enquiry should be regarded as immutable. In real re- 
search, investigators’ perceptions of the question they wish to answer alter 
as they learn more about it. My point here is that investigative managers 
need to reflect on the enquiry process and maintain a clear sense of their 
own purpose, how this is presented to, and perceived by, their informants, 
and the consequences in terms of the kinds of relationships established, 
and the information generated, in the enquiry. 


One of the most important elements establishing an investigator’s identity 
is the way in which she or he is introduced to informants. Most commonly 
this is through a ‘gatekeeper’ — someone in authority whose approval is 
needed before investigators can proceed to contact individual informants. 
The gatekeeper’s approval acts to legitimate and sponsor the enquiry. Typi- 
cally, gatekeepers are community leaders in rural areas, but could be com- 
pany managers or union officials in more industrial contexts. It may be that 
a gatekeeper is not hierarchically related to other informants, but merely 
agrees to introduce the investigator to potential informants in a social group 
of which the gatekeeper is a member (e.g. traders’ associations). 


Depending on the degree of hierarchy between the gatekeeper and the other 
informants, introduction and sponsorship by the gatekeeper may be a more 
or less important element in establishing a climate of collaboration for the 
investigation. In general, however, investigators are advised to be prepared 
to adjust the time and place of interviews to the convenience of inform- 
ants, and to explain the purpose of their enquiry. The collaborative climate 
between interviewer and interviewee is often referred to as ‘rapport’, and it 
is recognized that certain investigators have better possibilities for estab- 
lishing rapport with particular groups of informants than others, such as 
female investigators working with women’s groups. Differences of lan- 
guage, ethnicity, age, and class are also recognized as barriers to rapport 
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between investigator and informants. However, it should not be assumed 
that greater similarity between social background of investigator and in- 
formants is always conducive to better rapport. Classic ethnographic meth- 
odology seeks rapport by casting the researcher as the ignorant outsider 
seeking instruction in local history and custom. Similarly, informants may 
feel discussing sensitive issues with strangers from another culture poses 
less risk of what they say coming to the ears of neighbours and relatives. 


I think it is essential to be clear that establishing rapport is important both 
for structured questionnaires and for focused interviews, although particu- 
larly critical for the latter. Rapport should therefore be seen as a necessary 
part of effective interviewing. Further, this is so irrespective of the purpose 
of the research. Thus, although rapport may be an essential ingredient in 
participatory research which seeks to empower its informants, it may be 
equally critical for investigators who seek information from powerful groups 
in the hope that the findings can be used to restrict or reduce their power. 
My point here is that establishing rapport can be manipulative. Indeed, 
some researchers have suggested — and deplored the fact — that rapport is 
regarded as a form of social lubricant to elicit disclosure (Oakley, 1981). 
Establishing rapport is therefore not the same as empowering informants. 


For this reason, investigators using ‘participatory’ forms of enquiry such 
as PRA, discussed in Section 6.5, have a particular need for careful reflec- 
tion on the enquiry process and a clear sense of their own purpose in under- 
taking it. I believe that this ‘reflexive’ practice is important in maintaining 
investigators’ awareness of the power relations, to which I referred at the 
start of this chapter, inherent in any enquiry, and which can be masked 
where rapport creates the sense of an egalitarian relationship between in- 
vestigator and informant. Some formulations of participatory research which 
envisage outsiders taking part in enquiries only in a neutral, facilitating, 
role, for example, can be interpreted as implicitly denying the underlying 
power of the investigator in the enquiry process. This raises difficulties, 
firstly in the sense that denial of power suggests a denial of investigators’ 
responsibility in the enquiry. Secondly, as argued in the next section, it 
denies the ‘outside’ perspective that may be essential to investigators’ abil- 
ity to draw valid conclusions from the enquiry. 


I noted above that rapport is important in all interviewing, but it is espe- 
cially critical in semi-structured interviews which have the format of an 
informal conversation guided by the interviewer to cover the issues of in- 
terest to the enquiry, but free to pursue other unforeseen, but relevant and 
important, issues that arise during interview. General interviewing skills 
include the use of open-ended questions, probing (e.g. Who, what, when, 
where, why...?) to encourage more detail on points of interest, and explor- 
ing reactions to hypothetical (What if...?) scenarios. In addition, the ap- 
proach known as rapid rural appraisal (RRA; see also Chapter 7) brought 
together a variety of diagramming techniques which allow information to 
be set out visually. This has many advantages over purely verbal descrip- 
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tion in improving clarity (see Table 2.2 in Chapter 2 for a comparison of 
verbal and visual). It also generates a new focus of discussion, in which 
questions can be addressed to the diagram rather than direct to the inter- 
viewee. The principal types of diagramming are set out briefly in Box 6.2. 


BOX 6.2 IMPROVING COMMUNICATION THROUGH 


VISUAL TECHNIQUES 


Time line. A drawn line is used to locate different events in chrono- 
logical sequence. 


Activity/seasonal calendar. Variation in rainfall, work (agricultural 
work, migration), health (disease incidence, food scarcity), and other 
factors showing regular seasonal variations are recorded on calendar 
diagrams. The technique offers considerable scope for informants to 
create diagrams using their own calendars and systems of measure- 
ment, and identifying their own priority concerns. 


Transect. An interview conducted while walking a traverse of an area 
of interest to the enquiry. Allows the identification of different zones 
resulting from variations of topography, ecology, resource use, or 
ownership. The mobile interview prompts further exploration of the 
patterns of variation along the transect, and diagramming can be used 
to summarize the zoning pattern for further discussion. 


Venn/Institutional diagram. Circles are drawn or cut out of paper to 
represent different organizations or groups. Organizational relation- 
ships can be represented by the overlap between different circles. 


Mapping. May be used to map natural resources and resource use, 
ownership (e.g. of village land), residence, social differences, inci- 
dence of disease (e.g. within settlements). Mapping may be under- 
taken by different individuals or groups to visualize different percep- 
tions, or may be a public activity to encourage discussion and the 
visualization of a ‘consensus’ perception. 


Ranking exercise. These often use diagrammatic techniques to record 
the outcomes of comparing two alternatives (e.g. which tree species 
is best for firewood) through pairwise ranking, or a number of alter- 
natives through matrix ranking. Wealth ranking is undertaken through 
informants identifying wealth categories, recording each household 
on a card and then allocating the cards to the wealth categories. 


The use of diagramming techniques has been developed for interviews with 
individuals and with groups. In participatory rural appraisal, diagramming 
by groups has been developed into public activity in which informants are 
encouraged to identify their own questions and lines of enquiry (see exam- 
ples in Chapter 7). 
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6.5 PARTICIPATORY RURAL APPRAISAL 


Widespread advocacy of participatory forms of development has been as- 
sociated with the development of participatory methods of enquiry, of which 
participatory rural appraisal (PRA) is the best known. Originally devel- 
oped for work in rural communities, the approach has also been widely 
applied in urban contexts. Biggs (1995) has observed that such methods 
are fast taking on the form of a new generalized orthodoxy for solving 
development problems. Chapter 7 deals specifically with PRA, discussing 
particularly issues of power relations and PRA’s claims for empowerment. 
The concern of this section is to consider what issues PRA raises for devel- 
opment managers in relation to the rigour of enquiry — ‘Does it tell me 
what I want to know?’ and ‘Can I have confidence that what I have found 
out will not be contradicted or easily undermined by other, similar investi- 
gations?’. 


As a method of enquiry, PRA has the following characteristics: 


* It is firmly rooted in ethnographic traditions of semi-structured inter- 
viewing, and thus depends on the criteria for validity appropriate to semi- 
structured interview approaches. 

* Interviews commonly make use of diagramming techniques to construct 
visual forms of information, and this can be a powerful way of answer- 
ing ‘What?’ questions, as well as providing a focus for addressing ‘Why?’ 
and ‘How?’ questions. 

* PRA techniques are often used with groups in a public forum, with the 
aim of assisting all members of the group to contribute to a consensus 
response to the questions raised, whether in verbal or visual form. 


* Acentral concern of many proponents of PRA is that the enquiry proc- 
ess should encourage a process of empowerment: of informants in rela- 
tion to investigators, and of socially disadvantaged groups in relation to 
the prevailing power structure. This process of empowerment requires 
that investigators become ‘convenors, catalysts and facilitators’, encour- 
aging informants to take the initiative and set priorities for the enquiry, 
and that the enquiry ‘gives a voice’ to those whose opinions are not 
generally given prominence in the public arena. 


* PRA incorporates the idea that those participating should be able to use 
the enquiry as the basis for planning and implementing action to bring 
about change (one major network concerned with the methodology has 
renamed the approach ‘Participatory Learning and Action’). 

* Some proponents of PRA (notably Chambers, 1992) have counterposed 
‘participatory’ modes of enquiry, committed to empowering informants, 
to ‘extractive’ modes of enquiry which are guided by investigators’ agenda 
of ‘finding out’. 

Development managers who wish to interpret PRA activities as a means of 

understanding the situation in which they have to take decisions need to 
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consider the following points which have emerged from the experience of 
those using PRA approaches. 


Firstly, it is widely acknowledged that many PRA activities are undertaken 
by poorly trained ‘facilitators’ whose goals are limited to the production of 
physical outputs. Thus, the routine completion of a survey questionnaire is 
merely substituted by ‘doing a PRA’ — the routine (but ‘participatory’ ) pro- 
duction of village maps, calendar diagrams and other artefacts. It is impor- 
tant to recognize that such artefacts have no interpretive value unless ac- 
companied by an understanding of the context and process by which they 
were produced. (When, where, why, and by whom were they made? Who 
has seen them? What has been said about them since? And by whom?) 


A second, and fundamental concern in the interpretation of PRA has been 
identified by Mosse (1994, p. 510): 


... as public and collective events, PRAs tend to emphasize the 
general over the particular (individual, event, situation, etc.), tend 
towards the normative (‘what ought to be’ rather than ‘what is’), 
and towards a unitary view of interests which underplays differ- 
ence. In other words, it is the community’s ‘official view’ of itself 
which is projected... (O)ften the very structure of the PRA sessions 
— group activities leading to plenary presentations — assumes and 
encourages the expression of consensus. 


An example of this from the Mali study referred to earlier concerns about 
the lending of land to cultivators from outside the village. A census under- 
taken in landholders’ own homes revealed a much greater extent of land 
lending to outsiders than a public PRA mapping exercise which conveyed 
a ‘consensus’ that this was a relatively uncommon and unimportant prac- 
tice. 


Mosse argues that the public and consensus-building nature of many PRAs 
offers opportunities for dominant groups or people in authority to influ- 
ence public opinion so that their private interests become ‘officialized’ by 
incorporation in the ‘community consensus’. Conversely, the public arena 
sets considerable barriers for the expression of the views of groups — women 
in the case of Mosse’s work in India — who would normally simply not be 
expected to be present in formal public encounters, and who are unaccus- 
tomed to expressing themselves in terms suitable for a public audience. 


These observations raise the paradox that the more ‘participatory’ the en- 
quiry in the sense that it is undertaken in the public arena — following an 
agenda set by ‘the community’ — the more the outcome of the enquiry will 
mask the power structure of the community. Mosse (1995, p. 29) observes: 
‘... public participatory research methods are unlikely to prove good in- 
struments for the analysis of local power relations since they are shaped by 
the very relations which are being investigated.’ As a consequence, actions 
developed to meet ‘community goals’ identified in PRA risk reinforcing, 
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not changing, those power relations. From the point of view of develop- 
ment managers who wish to use PRA to understand social relationships as 
a basis for decision-making, this is a serious limitation of PRA. In terms of 
validity criteria, the consensus-building nature of PRAs restrict the expres- 
sion of difference and hence the scope for triangulation. Two important 
points flow from this. 


Firstly, investigative development managers using PRA must ensure ad- 
equate triangulation by talking to different types of informant, in different, 
less public social contexts. They need to reflect on and record the context 
and process of these encounters including their own influence on the infor- 
mation generated. Seeking out, interpretation, and use of information is 
guided by the investigators’ purpose in the enquiry. As Mosse (1995, p. 32) 
says: 


The often-used polarity between ‘extractive’ and ‘participatory’ 
research modes thus overlooks the fact that certain types of knowl- 
edge employed in participatory projects is [sic] necessarily external 
and analytical. Indeed, knowledge of social relationships which 
helps project workers identify the conditions for participation itself, 
to bargain with villagers on issues of equity, gender, or cost recov- 
ery ... is of this kind. [original emphasis] 


This need for investigators to establish their own understanding of social 
relationships in order to pursue their own purpose leads to a second point: 
the need to acknowledge that the investigator is an active agent and, no 
matter how participatory the enquiry, will affect its findings. This rein- 
forces the argument, made earlier, that investigative managers need to be 
explicitly aware of their agendas. If their purpose is to identify opportuni- 
ties to alter power relationships in favour of disadvantaged groups then 
investigators must keep this clearly in mind as they pursue the enquiry, 
irrespective of how they ultimately ‘negotiate’ this agenda with the local 
power structure. I noted earlier that the investigator as a purposeful change 
agent conflicts with the rather neutral-sounding role of ‘convenor, catalyst 
and facilitator’ allocated to outsiders in PRA. However, acknowledging 
this role and making explicit the investigator’s purpose is necessary to the 
rigour of the enquiry by allowing an adequate record of the investigator’s 
own influence on the context and process of the enquiry. 


6.6 CONCLUSION 


This chapter has set out issues which development managers will need to 
consider when finding out by talking to people. I have contrasted struc- 
tured surveys with semi-structured interviews, and emphasized the under- 
lying ground rules by which investigators try to improve the accuracy and 
reliability of information generated whichever approach is taken. I have 
argued that establishing a collaborative content is important whether or not 
the methods used are specifically labelled ‘participatory’. Those which are 
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so labelled, such as PRA, require similar criteria before they can be con- 
sidered rigorous and reliable as semi-structured interview approaches gen- 
erally. In this discussion I have begun to consider the power relations in- 
herent in encounters between investigators and informants and how these 
affect the confidence which development managers can allow themselves 
in the findings from such studies — questions dealt with more fully in the 
next chapter. 
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USING PARTICIPATORY METHODS 


HAZEL JOHNSON AND LINDA MAYOUX 


Many investigators concerned with policy development may want or be 
asked to use participatory approaches at some time. Participatory methods 
of investigation and policy development are increasingly promoted by 
development agencies of varying perspectives and influence, including the 
World Bank and non-govermental organizations (NGOs). In many north- 
ern aid agencies, participatory approaches are a required component for 
funding development programmes (Chambers, 1995a; Richards, 1995), In 
addition, the relationship between research and action is central to partici- 
patory investigation which is a reason for the appeal of such approaches to 
those involved in managing development. 


Investigators are also likely to be concerned with issues of empowerment, 
as empowerment is now an integral part of the policy of many develop- 
ment organizations. Given the explicit link often made between partici- 
patory investigation, participatory development and empowerment (e.g. 
Narayan and Srinivasan, 1994), it is important to know whether and how 
participatory methods can have an empowering effect, as well as under- 
standing their limitations and pitfalls. 


In Chapter 6 Philip Woodhouse questions whether participatory investiga- 
tion (specifically, participatory rural appraisal or PRA) can overcome in- 
equalities in power relations between investigators and their ‘informants’, 
and can also reveal the power structures within communities. The chapter 
concludes with advice to investigators to 


(a) triangulate their findings; 

(b) be aware of their own influence in the process of gathering informa- 
tion; and 

(c) keep in front of them their own agendas in carrying out an investiga- 
tion. 


In this chapter, we suggest some areas of investigation in which partici- 

patory methods of investigation for policy development can be potentially 

empowering, and discuss some of the limitations of trying to use participa- 

tory approaches as an empowering process. The chapter is concerned with 

the following questions: 

* Can a participatory approach to investigation act as an empowering 
mechanism? 


* How do power relations affect the use of participatory approaches to 
investigation? 
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© What are some of the key considerations for investigators and develop- 
ment agencies wanting to use participatory investigation as a means of 
empowerment? 


Section 7.1 examines the concept of empowerment and its presumed rela- 
tionship to participatory approaches to investigation and action. Section 
7.2 reviews some of the empowering experiences of using participatory 
methods while Section 7.3 looks at some of the pitfalls. Section 7.4 sum- 
marizes some of the critical issues which need to be addressed in using 
participatory approaches. 


7.1 PARTICIPATORY INVESTIGATION AND EMPOWERMENT 


Labelling a research process ‘participatory’ does not mean that it will auto- 
matically lead to ‘empowerment’. The language of participation and em- 
powerment is used differently by different people and has been appropri- 
ated by organizations with different ideologies and intentions. Unpacking 
the term ‘empowerment’ and reflecting on its relationship to participatory 
investigation are thus important tasks for any investigator concerned with 
the effects of research on participants or ‘informants’. The following re- 
flections can act as a starting point. 


7.1.1 WHAT IS EMPOWERMENT? 


Many researchers involved in policy development hope that their research 
will benefit those they are studying by enabling the views of marginal groups 
to be heard, and by stimulating debate within communities and develop- 
ment agencies. Whether participatory approaches to investigation can lead 
to the empowerment of participants depends not just on the conditions and 
processes involved, but on how the notion of empowerment is understood 
and used. Empowerment is a contested concept used in different ways by 
individuals and organizations of differing world-views and political per- 
suasions. There are disagreements about what is meant by power and hence 
the aims of empowerment, who is to be empowered, and the role of inter- 
vening agencies in empowerment processes. 


Power has different dimensions and operates in different, interrelated and 
often mutually reinforcing ways. For example, power generally has easily 
identifiable material dimensions, determining access to resources, levels 
of poverty and the range of choices and constraints affecting people’s abil- 
ity to achieve their aspirations. However, it also has more subtle manifes- 
tations in ideologies, values and discourses which might be shared unques- 
tioningly by those who are disadvantaged by them. Conformity to unequal 
social relations can be sustained in this way, and can have the effect that 
both the subordinate and the relatively powerful are protected from poten- 
tially threatening processes of change. Inequalities in access to resources 
and underlying or dominant values may reinforce each other at many ley- 
els of social interaction: between individuals, within households and fami- 
lies, in communities, and in national policy agendas as well as among in- 
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ternational organizations. For example, gender, age and ethnic discrimina- 
tion are complex mixes of material deprivation and ideological subordina- 
tion operating through many levels of social organization. Such interlock- 
ing relationships mean that it can be difficult to identify appropriate strat- 
egies for empowerment. For example, is it possible to address material 
inequalities without also changing underlying values which may be shared 
by those who are subordinated? Is it possible to challenge power relations 
among individuals, households and communities without also making 
changes in national and international structures and processes? 


In distinguishing the nature of different types of power relations, recent 
literature often refers to the difference between ‘power to’ and ‘power over’ 
(for example, Kabeer, 1994; Nelson and Wright, 1995), a distinction drawn 
by Lukes (1974). (This distinction is also made in Chapter 2, referring to 
Heron, 1989.) ‘Power to’ can signify the position of the relatively power- 
ful, but it also suggests a way forward for the relatively powerless: the 
development of people’s capacities to exert increased control over and/or 
to change their conditions of existence. ‘Power over’ signifies means of 
control over other people. In addition, some recent debates within NGOs 
have also stressed ‘power with’, a less individualistic form of ‘power to’ 
stressing alliances and joint action. An important issue for policy-oriented 
investigation and action is what sort of power (or empowerment) is being 
promoted (see Box 7.1). 


BOX 7.1 TYPES OF POWER RELATIONS 


‘power to’: ability to change the condition of one’s existence. 


‘power over’: control over other people. 
‘power with’: ability to achieve control through joint action with 
others. 


As is evident from the reflections so far, power is relational. However, 
while the idea of ‘power over’ suggests that the power of particular indi- 
viduals, groups or institutions only exists in relation to the greater or lesser 
power of others, and that an increase in the power of one implies a de- 
crease in that of another, the notion of ‘power with’ implies that increasing 
the power of some people does not necessarily diminish that of others 
(Rowlands, 1995, p. 102). Discussions of empowerment in development 
literature tend to refer to increasing the power of those disadvantaged in 
prevailing social relations: for example, the poor, women, children, old 
people, the disabled or particular ethnic groups. Understanding what sorts 
of power relations are involved in being relatively disadvantaged, and how 
the power of such social groups can be increased, is thus a common con- 
cern of development managers and other individuals and institutions en- 
gaged in policy change and implementation. This also makes investigatory 
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processes for policy development subject to potential conflict over the aims 
of investigation, how it is to be carried out, and what use is made of the 
outcomes. 


A consequence of viewing power as relational is that empowerment is an 
ongoing process, not an end state. Investigation and policy development 
may thus be designed to assist an ongoing process of increasing ‘power to’ 
as well as enhancing ‘power with’. Aspects of empowerment may also 
take many forms and occur in different social arenas, from individuals and 
groups becoming more articulate about needs and interests, to their being 
able to take control of particular areas of social action. Thus, in the exam- 
ples given in Section 7.2 below, some of the processes described have led 
to a growth in self-awareness or to an ability to articulate problems, while 
others have resulted in forms of negotiation over relations between differ- 
ent social groups. 


Thus if investigation is intended to assist policy development, it almost 
inevitably has to confront ways that power relations maintain the status 
quo, which they do by acting to resist change or influence the direction of 
change in particular ways. Power relations — including those in which in- 
vestigators are themselves embedded by their membership of institutions 
and their social position — also affect how investigation is carried out. These 
points suggest some key questions for investigators to reflect on at the 
outset of the investigative process: 


1 What are the aims and boundaries of the investigation? 


How should they be decided (and by whom)? Among whom and at 
what level is the investigation taking place (individuals, households, 
communities, etc.)? How might the population affect the investigation 
of power relations, and how might power relations in the population 
affect the investigation? Is the investigation concerned with ‘power 
to’, ‘power over’ or ‘power with’? 


2 Who is to participate in the investigative process? 


Is the aim to try and involve the participation of the relatively power- 
less or most disadvantaged? Is it possible to achieve it? How? How 
will that affect the investigative process as a whole? 


3 How will the relatively powerless or most disadvantaged participate 
and how can their participation be ensured? 


What would participation of disadvantaged people in a given investi- 
gation mean in practice? What would they be doing? What does it 
mean for the role of other participants and stakeholders? What does it 
imply for the role of the investigator? 


4 Howis the investigation to be linked to action? 


How can benefits to participants, including the relatively powerless, 
be ensured? How far can action be limited to particular levels (indi- 
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viduals, households, communities etc.)? Is action trying to change 
‘power to’, ‘power with’ or to challenge ‘power over’? 


5 What role is to be played by any intervening agency or agencies in 
both the research process and related action? 
What is the role of outside investigators and development agencies in 
influencing the design and aims of the research, the research process 
and outcomes? What are the decision-making procedures preceding, 
during and following the research? How does the agency expect to use 
the findings? To what extent is it able to learn from and change its 
policies as a result of the investigative process? 


7.1.2 PARTICIPATORY INVESTIGATION AND EMPOWERMENT 


In the late 1970s the philosophy, method and approaches known as rapid 
rural appraisal (RRA) were developed particularly by participants at a se- 
ries of workshops held at the UK Institute of Development Studies at the 
University of Sussex. RRA had its origins in: 

* a dissatisfaction with the biases of ‘rural development tourism’; 


* disillusionment with the normal processes of questionnaire surveys and 
their results; and 


* the search for more cost-effective methods. 


By the end of the 1980s RRA was being practised worldwide and in a 
number of different fields, particularly natural resource management, agri- 
culture and livestock rearing, credit programmes, health and nutrition. 
Despite its original name, it is now described as more relaxed than rapid 
and is not confined to rural areas. It has however historically been mainly 
associated with research institutes and aid agencies and ‘extracting’ infor- 
mation for development programmes. 


Participatory rural appraisal (PRA) has developed from RRA but with other 
influences, including those given in Box 7.2. One of the best known devel- 
opers and promoters of PRA, Robert Chambers, has described it as ‘a grow- 
ing family of approaches and methods to enable local ‘rural or urban’ peo- 
ple to express, enhance, share and analyze their knowledge of life and con- 
ditions, to plan and to act’ (1994c, p. 1253). This description suggests that 
participatory approaches to investigation not only allow for information 
gathering and interpretation which structured survey techniques are un- 
able to include (see Chapter 6), but that those ‘under investigation’ will 
also have some control over the investigation. They will be able to make 
their voices heard, increase their own awareness and understanding, and 
have a determining role in policy development which is directed towards — 
or is likely to affect — them. Such processes in turn call into question the 
role (and power) of the investigator (or investigating team) as well as of 
agencies which might be involved in designing and implementing policies 
and interventions. (Some of these issues are also raised in Chapter 2.) 
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BOX 7.2 PARTICIPATORY INVESTIGATION: 
SELECTED APPROACHES 

Activist participatory research (APR) 
This is a family of approaches and methods, inspired by the work and 
experience of Paolo Freire, which uses dialogue and participatory 
research to enhance people’s awareness and confidence, and to em- 
power their actions. Focusing on the underprivileged, and aiming at 
political action, it is based on a number of underlying principles: 
* that poor people are creative and capable and that they can 
and should do much of their own investigation, analysis and plan- 
ning; 
that outsiders have roles as convenors, catalysts and facilitators; 
that the weak can and should be empowered. 
Some of the techniques used include collective research through meet- 
ings and socio-drama, critical recovery of history, respecting and ap- 
plying ‘folk culture’ and the production and diffusion of new 
knowledges through written, oral and visual forms. 


Agroecosystem analysis (AESA) 


This draws on systems and ecological thinking and was developed in 
Thailand from 1978 onwards and spread outwards from Southeast 
Asia. It has contributed a range of key techniques: 


* transects (systemic walks and observation); 

* informal mapping (sketch maps drawn on site); 

* diagramming (seasonal calendars, flow and causal diagrams, bar 
charts, Venn or chapati diagrams); 

innovation assessment (scoring and ranking different actions). 


Applied anthropology (AA) 


In the 1970s and 1980s the numbers of social anthropologists in aid 

agencies rose, as did their status, which led to a greater appreciation 

of the importance of local knowledge. Some of the many insights and 

contributions have been: 

* the idea of field learning as a flexible rather than a rigid science; 

* the value of field residence, unhurried participant observation and 
conversations; 

* the importance of attitudes, behaviour and rapport; 

© the emic-etic distinction (building categories of investigation from 
the inside rather than imposing them from the outside). 


7 INVESTIGATION AS EMPOWERMENT: USING PARTICIPATORY METHODS 


Policy-oriented investigators work within an environment of complex power 
relations. They need therefore to be particularly aware of the implications 
of those power relations on the investigative process. For example, power 
relations within the field of study will structure the generation and control 
of knowledge both by individuals and by social groups: who can (or does) 
communicate with whom, about what, and for what reason; what types of 
knowledge are generated, who has access to them and how they are used. 
Research processes (and the use of research results) have the potential to 
increase, reinforce or transform these power relations, whether knowingly 
or unknowingly. Power relations between researchers, those who commis- 
sion them and those being researched affect the process and outcomes of 
investigation. 

This is true of all types of research, even those involving collection of 
‘factual’ data (see Chapter 11) or documentation. However, it is particu- 
larly tricky when trying to carry out rigorous investigation which involves 
‘thinking with people’, as outlined in Chapter 6. It is also an especially 
important issue when an explicit aim of investigation is the empowerment 
of disadvantaged people who have less control over the generation, access 
and use of knowledge. The relatively disadvantaged and powerless are usu- 
ally less able to translate knowledge into action, and are more vulnerable 
to the counter-actions of more powerful individuals and groups. 


As well as helping to understand better the social dynamics of power rela- 
tions, participatory investigation has been seen as contributing to empow- 
erment in a number of different ways. 
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1 Itis suggested that participatory approaches can help systematize local 
knowledge, increase awareness and enable groups which are not gen- 
erally heard to have more say in defining development needs and ac- 
tion. 


2 There is generally a strong emphasis on linking participatory in- 
vestigation and action (Nelson and Wright, 1995; Schrijvers, 1995). 
Schrijvers, for example, uses what she terms a ‘transformative approach’ 
which tries to bring about changes to benefit participants during 
the research process by siding with the concerns of those being 
researched. 


3 There is an emphasis on reversing power relations between investiga- 
tors and those being investigated. For example, Chambers (1994c) iden- 
tifies some important reversals in how investigative work is carried 
out in PRA: 


(a) reversals of frames: a shift from the categories and values of out- 
siders to those of local people, enabling them to help define the 
frame of investigation; 


(b) reversals of modes: the greater use of group work rather than indi- 
vidual informants; the use of visual rather than verbal techniques; 
and using comparison as a means of finding out quantitative data 
rather than direct measurement; 


(c) reversals of relations: establishing rapport and involvement with 
local people rather than reserve and distance from them; 


(d) reversals of power: enabling local people to enhance their own 
capacities for finding out and using and improving their own knowl- 
edge, rather than investigators extracting information for use else- 
where. 


These reversals can potentially change the ownership of investigative proc- 
esses and cycles of learning. Similarly, Schrijvers (1995) also combines 
her ‘transformative approach’ with ‘dialogical communication’: a process 
of exchange in which the investigator and the investigated are constantly 
changing places; an attempt to acknowledge explicitly, and thereby dimin- 
ish, the power of the researcher; and an attempt to agree objectives and the 
use of research results with those being investigated. You might compare 
these views with those outlined in Chapter 2 where it was pointed out that 
it is impossible to stand outside a ‘system’ under investigation in a so- 
called objective position. The importance of dialogue was also empha- 
sized. 


Having suggested points for reflection, let us now look at aspects of inves- 
tigatory practice which use participatory approaches. 
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7.2 POSITIVE EXPERIENCES OF PARTICIPATORY 
INVESTIGATION AND EMPOWERMENT 

The above reflections suggest a number of ways that participatory ap- 
proaches to investigation may contribute to empowerment. Among them 
are their potential to uncover social relations between the more and the less 
powerful, the reversal of relations between investigators and investigated, 
and the close link between investigation and action. This section first looks 
at three areas where participatory methods of investigation have been seen 
as particularly relevant to empowerment processes — poverty and liveli- 
hoods, health, and raising awareness for action — while the next section 
examines some of the problems of these assumptions. 


7.2.1 POVERTY AND LIVELIHOODS 


Several careful comparative studies have shown that well-conducted, par- 
ticipatory methods can be more reliable than conventional surveys in un- 
covering dimensions of social differentiation and the structures and rela- 
tions of local poverty. In principle, participatory methods do not impose 
categories of analysis but try to elicit them from otherwise often ‘invisible’ 
groups such as elderly people, poor women or children. It may be easier to 
get members of such groups to participate in a group discussion than for 
each of them to fill in a questionnaire or be interviewed separately — they 
may be illiterate, or have more powerful partners, carers or others who 
would either claim to speak for them or prevent them taking part in a sur- 
vey independently. Through PRA methods, for example, local wealth 
rankings can take into account a wide range of considerations such as types 
of ownership of land and livestock, types and amounts of debts and repay- 
ment capacity, types of job, whether permanent or temporary, capacities to 
educate children, and so on (Chambers, 1994c). 


Eliciting such information by using participatory methods can be relatively 
quick (and cheap) compared with other methods of research. Chambers 
(1994c) relates how in Kabripathar village in Gujarat an illiterate woman 
from a neighbouring village facilitated census mapping by women on to 
cards, leading to a full village census of 87 families, giving numbers of 
women, men, girls, boys, bullocks, cows, buffaloes, goats, donkeys and 
other information. This process was completed and checked in about four 
hours. Seasonal calendars and flow diagrams have also been rapidly elic- 
ited from groups of poor and illiterate children, women and men showing 
important information which would otherwise take a long period of field 
research to gather using conventional methods. 


Participatory investigation can be useful in challenging the agendas set by 
outsiders, or for discerning whether one dimension of need can be seen in 
isolation from other economic and social factors. For instance, in one piece 
of research in Bangladesh, the main concern of the researchers was health. 
However, a problem priority matrix ranking carried out with landless women 
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showed a different set of concerns, which are shown in Table 7.1 (from 
Welbourn, 1992, p.6). The women were asked to name their main prob- 
lems. Symbols for each of the problems mentioned were drawn on sepa- 
rate cards. The women were then asked to compare the problems in pairs, 
until an overall sequence of importance was established. The results showed 
that the women’s overriding concerns were for means of livelihood, the 
future reduction of access to natural resources (the river bank on which 
they lived), and lack of business opportunities. 


This is a result which might have been obtained using a survey. The par- 
ticipatory ranking method was quicker and did not rely on literacy. In ob- 
taining a consensus view through discussion this method may have tended 
to downplay differences in priorities between different women. However, 
interesting insights into the reasons for such differences in the women’s 
priorities might have occurred during these discussions (which would not 
have occurred if individual rankings had simply been combined analyti- 
cally by the researcher). 


Participatory approaches have also been used successfully to reveal con- 
siderable differences between people in ways of categorizing and analys- 
ing both physical and social phenomena. For example, in South India, a 
study showed that both men’s and women’s criteria for poverty were ‘lack 
of economic opportunities’. However, for men, this meant lack of employ- 
ment options, while for women a key dimension was whether their male 
family members would allow them to take jobs if they were available 
(MYRADA, 1992). Mapping exercises have found significant differences 
by gender and age in representations of physical space and access to local 
institutions. Figure 7.1 shows an example given by Welbourn (1991) of 
two mental maps from a village in Sierra Leone, one drawn by men and 
the other by women. The men’s map includes all the roads which lead out 
of the village whereas the women’s map concentrates more on the village 
centre. The men have identified boundaries and objects of status, such as 
the cotton trees, the (broken) chairman’s tractor, the village drum, while 
women did not show any of these. When asked to mark changes they would 
like to see, men marked a series of important buildings lining the route of 
the main road leading to the village, including an administration hall near 
to the centre. However, when the women were asked to mark changes, 
they first explained that this was not their role: ‘women do not have any 
power to decide where any of these things should be. The men have the 
last say’ (Welbourn, 1991, p. 17). When they were encouraged to imagine 
changes, they drew a huge hospital, close to the centre, followed by a 
school and wells to serve them. In another, poorer village, when women 
were asked to draw changes on their maps, they replied ‘we can’t draw 
changes on this map, because the kind of changes we need can’t be drawn’ 
(ibid). The unmappable issues they were concerned about were overwork, 
breakdown in co-wife support and beatings from their husbands. 
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FIGURE 7.1 Men’s and women’s village maps in Sierra Leone 
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7.2.2 HEALTH 


Participatory methods can also be used to investigate sensitive areas essen- 
tial to the success of health programmes. One example is in relation to 
women’s knowledge of reproduction, often a taboo or mysterious area but 
vital to increasing women’s control over their lives. Interviews and body 
mapping techniques have been used in Zimbabwe to examine women’s 
knowledge about reproduction and non-indigenous contraceptive methods 
(Cornwall, 1992). Interviews were used to find out what women had been 
told about contraception by different sources and what they thought about 
oral contraception. Maps of reproductive organs were drawn by women 
(individually or in groups) on paper or on the ground (see examples repro- 
duced in Figure 7.2). After an initial discussion of their body maps, women 
were asked to draw where conception took place and describe the proc- 
esses from fertilization to birth, referring back to their maps. Using these 
maps, researchers were able to work together with the women towards 
explanations of contraception and ways of preventing conception which 
were locally appropriate. For several women in Cornwall’s study this was 
the first time that they had drawn anything, and for most of them, it was the 
first time they had thought about their internal anatomy in such a way. Use 
of visual methods was useful to help clarify ambiguities (for example in 
understanding the term ‘womb’ ), and provided a shared reference point for 
participants and researchers. The maps increased women’s confidence in 
their own knowledge and also served to highlight some of the researchers’ 
own biases by contrasting the biomedical frame of reference of the re- 
searchers with the ideas about human biology held by the women. 


Participatory methods have also been used to investigate how power rela- 
tions have affected the outcomes of health policy. For example, LaFond 
(1992) describes participatory research looking at the reasons for failure of 
an immunization programme against a range of diseases (including mea- 
sles, tetanus, polio and whooping cough) conducted by the Ministry of 
Health in Somalia. There was good initial take-up, which diminished in the 
second and third round of vaccinations despite a dramatic decline in some 
diseases. Participatory research revealed that women’s reactions to the 
immunization programme were part of complex social relations 
surrounding local practices and people’s connection with the state struc- 
tures implementing health policy. The investigative team found that mes- 
sages used to promote acceptance of the immunization programme did not 
reflect mothers’ perceptions of certain diseases and failed to inform, edu- 
cate or motivate them. For example, mothers made a distinction between 
neo-natal tetanus and other tetanus and gave them different names and 
causes. Thus health messages about neo-natal tetanus were not interpreted 
correctly because the local term was not used. In addition, neo-natal 
tetanus was seen as having spiritual causes and although there was a reduc- 
tion following immunization, the causal connection was not made. Thus 
few mothers believed that immunization could prevent neo-natal tetanus. 
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FIGURE 7.2 Body map diagrams (Cornwall, 1992, p. 71) 


Mothers also feared side effects from tetanus immunization and spread 
rumours that it would cause infertility. These attitudes were exacerbated 
by the promotion methods of officials who were concerned to link immu- 
nization with political goals. The researchers suggested that trusted tradi- 
tional healers should be involved to facilitate discussion on immunization. 
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In this instance, adopting a facilitative approach meant that an investiga- 
tion could be carried out in a politically sensitive situation in which meth- 
ods such as formal surveys might have aroused opposition on the part of 
government officials. 


7.2.3 SOCIAL AWARENESS FOR ACTION 


Participatory methods can be used to raise awareness of inequality and for 
lobbying. For example, visual PRA material has been used by NGOs such 
as MYRADA in India to investigate various aspects of gender inequality: 
women’s access to information, decision-making in households, women’s 
access to institutions (see Figure 7.3 for a visual representation) and 
interlinkages between different dimensions of gender subordination. A par- 
ticularly innovative example is the use of PRA in Gujarat to raise men’s 
awareness of gender subordination (Bilgi, 1992). Men had resisted the in- 
troduction of labour-saving devices such as biogas, pressure cookers and 
flour mills because they found the food produced by these methods taste- 
less and were not particularly concerned to reduce women’s workloads. 
PRA methods were used to raise men’s awareness of women’s work and 
enable them to reflect on their initial perceptions. The men then went on to 
look at the nature of the women’s tasks and to evaluate how ‘hard’ they 
were compared with their own. 


self-help 
group fair price shop | VAO 
hospital ~_| cyte 
school <—____| 
MEN police 
> bank « > court 
MYRADA 
office and Roar 
tat BDO 
la village level worker 
note: 


(1) MYRADA works only with women in this village. 
(2) Women mainly visit institutions located in their own village. 


(3) Men's visits to outside institutions are combined with other work they may have 
outside, ¢.g. marketing. 


key: 
VAO is village administration office 
BDO is block development office 


FIGURE 7.3 Drawing of institutional linkages by gender in an Indian village 
(MYRADA, 1992, p. 11) 
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PRA can also help to facilitate decision-making processes for action after 
an investigation has taken place. For example, Shah and Shah (1995) re- 
port a project where lift irrigation was being proposed. An appraisal pro- 
cess revealed that with only a very small additional investment the irri- 
gation pipes could easily be extended to cover the small homestead plots 
used by women to grow one rainfed crop of vegetables and a little maize. 
The women were confident that with a little water they could grow veg- 
etables all year round. Members dug the channels for the pipes and the 
women had access to the irrigation water at the same time as men who 
used it for their main agricultural fields. This led to conflict as men argued 
that their main crop was more important than the vegetables and other 
crops grown by the women on their homestead plots. They suggested that 
women should not be allowed to irrigate their plots until the men had irri- 
gated all their fields. Because they had no bargaining power, most of the 
women lost their winter vegetable crop in the first year. The women then 
decided to withdraw their membership from the irrigation co-operative. 
However, with support from the NGO, a compromise was reached, in which 
women members would have access to the irrigation water provided that 
they all used it on the same day to minimize time and water loss. Other 
examples cited by Shah and Shah which have resulted in greater aware- 
ness and negotiation over new social arrangements include: giving disad- 
vantaged groups a higher share in employment, ground water budgeting 
and management, allocation of surplus common land, forest protection and 
management of buffer zones in national parks. In many cases these 
processes involved conflict in the short term but successful negotiations 
over longer-term agreements (Shah and Shah, 1995, p. 49 and p. 51). 


7.3 SOME LIMITATIONS AND PITFALLS OF PARTICIPATORY 
METHODS 


It is important not to idealize the empowering capacities of participatory 
methods. Although there are other limitations, of particular interest are that 


(a) increasing participation does not automatically make it easier than with 
other methods to grapple with power relations within the boundaries 
of research; 


(b) inequalities cannot necessarily be addressed by the research process; 
and 


(c) it is difficult to ensure empowering outcomes. 
We look briefly at examples of these limitations. 


7.3.1 POWER RELATIONS WITHIN RESEARCH BOUNDARIES 


As Chapter 2 has suggested, it is important to understand the nature of 
power relations within the boundary of the investigation. However, par- 
ticipatory methods alone may be insufficient to do this. One example is 
provided by Mosse (1995; see also Chapter 6). PRA methods were used in 
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western India to establish basic socio-economic and agro-ecological knowl- 
edge for village-level planning and implementation and to identify priority 
problems in a participatory farming systems development project. Research- 
ers were also interested in how PRA could help ‘outsiders’ to understand 
the social context of their initiatives sufficiently to work effectively within 
local communities. In the initial stages local participants tried to promote a 
view of community harmony to outsiders, while in practice powerful mem- 
bers of the communities attempted to sabotage the networks used for PRA. 
In addition, gender relations in the villages made the participation of women 
difficult. Although it was possible to address some of the problems over 
the longer term, Mosse (1995, p. 32) says that this was not possible through 
PRA alone: 


The essential material for our social analysis is not found so much 
in the agreed output (on map, chart or diagram), but in the ab- 
sences, the gaps and corrections, the after-thoughts, the errors and 
false-starts, the disagreements or conflicts, even the complete 
failure of a PRA exercise. Through these we get glimpses of how 
power operates in the community ... [However] much of this social 
information, like that from observing activities, did not arise from 
participatory appraisal. It was not a type of ‘people’s knowledge’. 
Indeed, these insights (in some cases represented by field workers 
in their own analytical Venn or chapati diagrams of villager groups, 
factions, alliances etc.) were not, and probably could not have been, 
generated in group discussions by villagers. Like other analytical 
models, and like the knowledge gained from the critical reflection 
on project activities, they represented an external viewpoint ... Of 
course, local people already have the sophisticated knowledge 
necessary for everyday social life. Often this knowledge remains 
tacit and need not, or cannot without risk of conflict, be made 
explicit. The often-used polarity between ‘extractive’ and ‘partici- 
patory’ research modes thus overlooks the fact that certain types of 
knowledge employed in participatory projects is necessarily exter- 
nal and analytical. 


7.3.2 INEQUALITIES IN THE RESEARCH PROCESS 

Participatory approaches can still exclude people from investigation, par- 
ticularly the very poorest. In practice, ‘key informants’ are often used. 
Welbourne notes, for example, that wealth rankings carried out in Malawi 
commonly found that the people who spoke in meetings and showed re- 
searchers around were generally those who had more respect and self-con- 
fidence and were from relatively high status social groups. They presented 
their own problems, outlooks and needs. By contrast, the richest people 
did not attend meetings or show researchers around because they had little 
to gain and had their own means of access to resources, while the poorest 
people did not attend meetings because they did not think that the proc- 
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esses would be relevant to them (Welbourn, 1991). Minorities or groups in 
danger of violence are particularly unlikely to participate. Research by 
Mosse has also shown that women often do not participate unless they are 
specifically targeted and encouraged (Mosse, 1995). It can also be difficult 
to get children to participate and sustain their interest (Guijt er al, 1994; 
Baker, 1996). Even in carefully targeted and differentiated exercises, it 
cannot be assumed that the poorest are being reached. For example, a map- 
ping exercise in a low caste area in Tamil Nadu missed out some house- 
holds. On talking to the large number of local women and men involved, 
they stated that the missing households were ‘only huts, we didn’t count 
those, they are the downtrodden who depend on daily labour’ (Pretty et al., 
1992, p. 42). 


7.3.3 DIFFICULTIES IN ACHIEVING EMPOWERING OUTCOMES 


In many cases the strength of power relations may make it impossible for 
participants to act directly on needs and policy issues identified in the in- 
vestigative process. However, participants may find indirect ways of tak- 
ing action. For example, SARTHI (Social Action for Rural and Tribal In- 
habitants of India) conducted research in Gujarat on women’s health and 
empowerment. It transpired that women had long shared information about 
their husbands’ drinking habits and the subsequent wife beatings. How- 
ever, the women decided not to lobby for closing the drinking houses be- 
cause they felt they would not be able to cope with their husbands’ reac- 
tions. Instead they concentrated on other, more acceptable, activities which 
helped to strengthen their position (Renu Khanna, personal communica- 
tion quoted in de Koning, 1995, p. 36). 


Who participates in the investigative process is also likely to influence 
outcomes. Where it is not carefully targeted, participatory approaches may 
be used to legitimize the demands of the more powerful. This was a prob- 
lem encountered by the Mosse study cited above. Another example is given 
by Richards (1995) of a study of the Mende in Sierra Leone. It describes 
how village elders created space for people to express differences of opin- 
ion in public meetings. However, the elders’ management of this space 
ensured that the decisions made generally favoured the village elite. 


Even when decisions are not made in the interest of powerful vested inter- 
ests, it is likely that they will be oriented towards the interests of those who 
have been most active in the research process. While this may not be a 
‘pitfall’ as such, it is important to be aware of the implications. For exam- 
ple, Ison (1996) describes using an action participation research approach 
to investigate why graziers in New South Wales, Australia, were not adopt- 
ing new practices to limit rangeland degradation. The project attempted to 
construct and constantly negotiate shared meanings and understandings of 
what it meant to be a pastoralist, and of the practices and problems of 
rangeland farming. Using a six-person team over nine months, graziers 
were given semi-structured and in-depth interviews and then were invited 
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to attend joint discussions, become co-researchers, and present and discuss 
particular ‘enthusiasms’ for action. At this point, the main difficulty for the 
researchers was to stand back and allow the graziers to identify points of 
action. In addition, although the graziers came with a diversity of 
‘enthusiasms’, the process of arriving at consensus for action resulted in a 
relatively small number of graziers carrying it forward. Those who did 
continue to be active were the graziers whose ideas had become the con- 
sensus. 


Participatory approaches may also expose and reinforce rather than ad- 
dress conflicts of power relations. Such tensions can be a particularly dif- 
ficult area for development managers and development agencies to handle. 
For example, Shah and Shah describe the consequences of a participatory 
appraisal of natural resources in a village in Gujarat. The investigation 
showed that a low caste group, the Gadvis, had little access to surface 
water resources. A high caste group, the Rajputs, owned the large fields 
and controlled most of the common property resources, including water. 
Both groups wanted wells dug on their land and NGO engineers identified 
suitable sites. The Gadvis raised the capital required and the group started 
constructing its community well. They developed a land use plan and started 
preparations for cultivating in the winter season. However, in the process, 
the Rajputs had lost their source of cheap wage labour in the Gadvis, be- 
cause the Gadvis had previously needed to do wage work to supplement 
their low incomes from their unirrigated fields. Several meetings were or- 
ganized between the groups during this period for negotiations and bar- 
gaining. But while the Gadvis were still working on their wells, they were 
ambushed and beaten up by a group of Rajputs. Two of the Gadvis died 
and others were injured. The NGO facilitators had not been able to contain 
the effects of the participatory process of investigation on this power strug- 
gle. Rather than initiating further programmes, the NGO has since spent 
more time facilitating negotiations between the two groups (Shah and Shah, 
1995, p. 50). 


7.4 INVESTIGATION AS EMPOWERMENT: CRITICAL 
CONSIDERATIONS 

Although participatory methods of investigation can act as empowering 
mechanisms under certain circumstances, some of the pitfalls are also clear. 
The contributions and pitfalls we have mentioned are summarized in Box 
7.3, but more can be found in the now extensive literature (see Box 7.4). 
What is evident is that using participatory methods of investigation as a 
means of empowerment requires a high level of awareness on the part of 
investigators and their institutions as to what the investigative process is 
for. It also means being highly aware of researcher preconceptions and 
bias, of the link between investigation and action, and of ethical considera- 
tions. We now look briefly at these aspects. 
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BOX 7.3 INVESTIGATION AND EMPOWERMENT: POTENTIAL 
CONTRIBUTIONS AND PITFALLS OF PARTICIPATORY METHODS 


Potential contributions 

* Can help to break down barriers beween investigators and investi- 
gated. 

* May empower by raising awareness and developing skills. 

* May catalyse discussion of strategies for change. 

* May be an important way of integrating the poor and disadvan- 
taged into decision-making. 

Potential pitfalls 

* May be used by the more powerful to legitimate their own de- 
mands. 

* May expose, but may not deal with, conflicts. 

* May come up with a ‘shopping list’ of demands with little guide as 
to how these can be prioritized, 

* May give illusion of ‘empowerment’ which may be token and not 
integrated into decision-making. 


BOX 7.4 FURTHER SOURCES ON PARTICIPATORY INVESTIGATION 


Continually updated bibliographies on participatory investigation and 
action can be obtained from the Institute of Development Studies, 
Sussex University, UK. The International Institute for Environment 
and Development (ITED) in London, UK, has published a number of 
manuals as have many international aid agencies such as the World 
Bank (e.g. Narayan and Srinivasan, 1994, above). A list of aid agen- 
cies currently conducting PRA can be found in Chambers, 1994b or 
1995b. Addresses of these organizations and other centres for partici- 
patory research in the South can be found in Eade and Williams (1995). 


7.4.1 FOCUSING ON PROCESSES AS WELL AS OUTPUTS 

One area of awareness involves being clear about the purpose of the re- 
search. Although never an easy task, this is probably more straightforward 
using conventional research methods than it is when the research process 
is intended to include other people in the definition and design of it. How- 
ever, participatory approaches can have advantages over other research 
processes which might meet obstacles at a later stage if local social 
dynamics and relations have not been taken into account earlier on. 
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In participatory investigation, there is also a difficult balancing act between 
research process and obtaining research outputs. Investigators might have 
a set of concerns or an agenda which requires specific outputs — for exam- 
ple, finding out about local healthcare needs to develop a healthcare policy 
for their organization. However, the concerns of other participants in the 
research process may be different (for example, as in the priority ranking 
exercise illustrated in Table 7.1). If, however, investigators also want to 
enable people within the boundary of the investigation to have a say in 
how the research is carried out and to be able to influence policy and action 
outcomes, decisions will have to made about whose agendas should have 
priority or how different agendas can be combined. 


The tension between process and output in participatory investigation can 
also affect the time required to carry out such research. Although particular 
methods can produce rapid results (as in the Chambers example in Section 
7.2 of doing a village census in a few hours), the time between the begin- 
ning of the investigative process and outcomes can also be quite extended 
(as Ison’s nine month study of graziers, mentioned in Section 7.3). This 
means that there is a temptation to concentrate on the output by using a 
semblance of participatory approaches rather than actually allowing the 
investigative process to have the time and depth it may require to enable 
participants to gain investigative or organizational skills (‘power to’) and 
have more control over outcomes (‘power over’ or ‘power with’). 


Steering an investigation which is intended to have an empowering effect 
on participants in a multiple-actor situation suggests that there are likely to 
be conflicts — or, at least, differences — of values over the purpose, process 
and desired products. To be an effective investigator in these circumstances 
requires a number of skills such as: the ability to sustain a dialogue across 
different social groupings about their needs and interests; the ability to 
negotiate; and the ability to build consensus about process and outcomes 
in order to get something done. An equally important task could also be 
enabling other participants to gain such skills. However, the investigator 
must also try to steer a research process through the concerns of other 
stakeholders or interested parties such as other organizations working in 
the area, government ministries, international organizations, etc. Knowl- 
edge and understanding of the institutional arena and its dynamics is nec- 
essary to inform reflection on the research goals, processes and outputs 
(Chapters 5, 8 and 10 address this point in different ways). As a final point 
investigators also need to be highly aware of and to manage their own 
‘power relations’ — both with respect to other participants and stakeholders 
in the research process, and to their own position in their organization. 


7.4.2 PRECONCEPTIONS AND EXCLUSIONS 


The construction of boundaries — which is a necessary part of the investi- 
gative process — must be carried out with a great deal of care in participa- 
tory research. (The concept of boundaries is explored in detail by Blackmore 
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and Ison in Chapter 2.) Care is needed not only in deciding what is and is 
not within the investigative boundary but also in the focus or course of 
inquiry within it. Questioning assumptions about boundaries is therefore 
an essential part of the investigative process, especially if particular cat- 
egories of people are not to be excluded. An obvious example is the now 
commonly debated concept of household. Households may be seen as sin- 
gle units of analysis by both participants and facilitators; for example, in 
investigating poverty and livelihoods, However, the composition of house- 
holds, what differences exist between members’ access to and control over 
resources and over decision-making, how perceptions of the roles of the 
old, the young, men and women are constructed, might all be important 
dimensions in investigating and understanding a given set of livelihood 
issues. 


People can be excluded from participation because of the greater influence 
of powerful groups within communities, or because of the interests, agen- 
das and world-views of other organizational stakeholders. Some of the case 
studies mentioned above revealed different problems of this kind in the 
dynamics of investigation: resistances among different groups within com- 
munities; different perceptions of needs and the purpose of investigative 
activity; control of some groups in a given population by others; attempts 
to direct or even sabotage the investigative process. 


How can investigators try and ensure that the people relevant to a particu- 
lar investigation take part? Part of the answer might lie in greater aware- 
ness of one’s own and others’ preconceptions of social relations and dy- 
namics — and encouraging dialogue and debate about firmly-held ideas. 
However, it may also involve setting up checklists directed to the needs of 
a specific investigation. For example, Dent (1996) has suggested points to 
establish whether women are being included or excluded in the investiga- 
tive process: 


* Timing: are women able to join in the activities? 


* Location: where are women allowed to meet? Where will they feel com- 
fortable and able to speak? 


* Representation: are women represented in the PRA team? 


© Relevance: are issues covered which are relevant to women? Are there 
issues relevant to some women which are excluded? 


* Equal roles: are women able to have an equal voice in mixed meetings, 
perhaps with the facilitator’s help? Or will separate meetings for men 
and women be more effective? 


* Marginalized women: are minority groups of women, such as widows, 
divorcees, disabled, and the elderly, seen and heard? 


(Dent, 1996, p. 22) 
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Such questions could be used as the basis for thinking about the participa- 
tion of other subordinate groups. 


7.4.3 INVESTIGATION AND ACTION 


The reality of much policy development is that, at least in the initial stages, 
information is required quickly and is often incomplete, even if informa- 
tion gathering over the longer-term can be set up through monitoring pro- 
cedures. Given the often finite periods of time which any investigator or 
worker for government or other organization is likely to spend in a particu- 
lar site of investigation, follow-up processes can only be seriously pursued 
if there is local control over and desire for continuous knowledge-building 
— that is, the empowerment process. However, taking into account such 
considerations is not always easy as an example from Schrijvers (1995) 
illustrates. 


Schrijvers carried out a three-month research project exploring needs per- 
ceived by Tamil and Muslim refugees in Sri Lanka. She took a 
‘transformative approach’ (see Section 7.1.2) which involved siding with 
the concerns of the refugees. She reflected on some of the contributions 
and limitations of her study. Contributions of the investigation included 
the fact that the refugees she worked with were listened to in ways they 
had not experienced before. Knowledge, views and perceptions were shared, 
and Schrijvers was able to construct an analysis of the violence affecting 
the refugees as well as help them to identify ways in which they could 
protect themselves. The research was also used for advocacy and action to 
resist the further displacement of a particular group. On the downside, 
Schrijvers pointed out that the time was too short, language was a barrier 
to communication, and because the refugees were a heterogenous group it 
was difficult to establish shared objectives and desired outcomes for the 
investigation. In addition, after she had left the research area, the possibili- 
ties for sustained investigation and action were limited. 


This example suggests that an institutional base for linking investigation to 
action is key. For example, Narayan and Srinivasan (1994, p. vii) state: ‘in 
order to have significant impact, participatory approaches must be ac- 
companied by institutional reform supportive of local initiative’. Thus, in a 
worst case scenario, an isolated piece of participatory investigation may 
raise expectations among participants about future action which cannot be 
realized by investigators or their organizations. However, assuming an in- 
stitutional context which is favourable to local action, participatory inves- 
tigation can in principle provide a basis for ongoing dialogue about the 
nature and direction of policy development. 


Even given a favourable institutional environment, who takes action and 
on what basis is also open to question. For example, there may be a choice 
for an external investigator between standing back and allowing other par- 
ticipants in the investigative process to determine what action should be 
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taken (for example, in Ison’s New South Wales graziers case above), or 
ensuring that an external agency takes responsibility and intervenes. In- 
stances when it is extremely important to do the latter include conditions 
of extreme social conflict. 


7.4.4 ETHICAL CONSIDERATIONS 

As we have illustrated, it would be naive to assume that participatory in- 
vestigation will necessarily be empowering for participants. Rather than 
empowering people, participation can make them extremely vulnerable, if 
sensitive information enters the public arena, or if knowledge and under- 
standing which enables marginalized people to cope with difficult social 
situations is appropriated by others already more powerful than themselves. 
Such possible effects place a high level of social and ethical responsibility 
on those initiating and supporting the investigation. 


How rights and responsibilities are to be established in the investigative 
process is a crucial consideration for those carrying out participatory re- 
search. This is particularly important in the ownership, and thus acknowl- 
edgement and use, of research output, as well as in issues of openness and 
confidentiality. Although ethical concerns apply to all forms of research, 
they are brought even more sharply into focus by the anticipated reversals 
that participative approaches are expected to bring. 
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8 ORGANIZATIONAL ASSESSMENT AND 


INSTITUTIONAL FOOTPRINTS 
CHRIS ROCHE 


This chapter concentrates on how one might ‘find out fast’ about a variety 
of organizations in the public, private and non-profit sectors, for a variety 
of purposes. One might, for example, want to make a decision about whether 
to support an organization, or ally with it, or oppose it. One might want to 
know how to influence an organization, or what part the organization itself 
plays in public action to influence policy choice. 


Following the definition used by Dréze and Sen (1989), I take public ac- 
tion to mean not only the activities of the state, but also actions by organi- 
zations of citizens, whether collaborative or adversarial in nature. This 
approach goes beyond recognizing that government policies are a result of 
the accumulated response to a variety of interests within government and 
from pressure groups and individuals outside government. It also recog- 
nizes that other organizations work alongside government agencies to pro- 
duce policies and actions aimed at public benefit. So it is important to find 
out how their institutional structures, procedures and actors represent and 
further particular interests. 


In general, the point of any policy-oriented investigation of an organiza- 
tion will be to promote change. This could be change in the organization 
itself resulting from an organizational assessment, or it could be public 
action on the part of an organization informed by an investigation of how 
another organization contributes to public action. These two cases form 
the substance of Sections 8.4 and 8.5 below. The second case involves the 
idea of ‘institutional footprints’. An organization (or institution) leaves 
‘marks’ where it has been active, which can be studied to give ‘clues’ both 
about the organization itself and the effect of its actions. 


Before looking at these two cases of how investigating organizations can 
inform policy change, there are three more general sections. Section 8.1 
introduces some different theoretical perspectives on how organizations 
can be understood. Some common questions emerge from these, which 
are outlined in Section 8.2. Section 8.3 then explores preliminary ways of 
answering these questions, using grey materials and informants both in- 
side and outside the organization — to set up a framework for a more fo- 
cused investigation aimed either at organizational change (Section 8.4) or 
at promoting change in the policy environment (Section 8.5). 


8.1 WAYS OF LOOKING AT ORGANIZATIONS 

This section refers to a selection of authors who describe different ways of 
looking at organizations. Their multiple perspectives help in developing a 
rounded understanding of public action and in identifying some common 
and fundamental questions. 
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Gareth Morgan, in his book Images of Organization (Morgan, 1986), ex- 
plains that organizations are complex, contradictory and paradoxical. They 
are many things at once. Morgan draws on different disciplinary traditions 
and uses ideas such as ‘organization as machine’, ‘organization as organ- 
ism’, ‘organization as political system’, ‘organization as culture’, etc. He 
suggests that looking at images and metaphors of organizations in this way 
allows us to take account more consciously of what we already know intui- 
tively — that where you sit and who you are influences how you see an 
organization. 


Building on this idea, I will explore three different frameworks you can 
use to analyse the role of organizations in public action: 

1 the motivation of different stakeholders; 

2 the organization’s ‘gender archaeology’; 

3 organizations and change. 


8.1.1 STAKEHOLDERS, MOTIVATIONS AND INCENTIVES 

The motivation, incentives and rewards of different stakeholders within an 
organization, and also the pressures that are exerted on them from the 
broader environment, will shape the organization’s actions. So organiza- 
tions can be viewed ‘less as products of culture, tradition, and history and 
more the result of the interactions of sets of rational actors pursuing mate- 
rial goals’ (Moore et al., 1994). 


This raises the questions: 

¢ What encourages staff to work towards making an organization per- 
form? 

© What are the sources of the pressures to do so? 

Moore et al. (1994) suggest that there are perhaps up to seven sources of 

pressure in the broader environment, which influence outcomes for any 

organization. For some organizations, particular stakeholder roles might 

not exist, while for others they may be combined. These roles are: 


1 Owner: for private companies, the shareholders; for membership or- 
ganizations, the paying members; less clear for non governmental or- 
ganizations (NGOs) or government departments. 


2 Financier: can take at least three forms: 
* investors who expect a return; 
* grantors who give funds for a particular purpose; 


* dispersed donors who give amounts towards the organization as a 
whole. 
3 User: a private company’s customers; owner-users in membership or- 
ganizations; those benefiting directly or indirectly from the services the 
organization provides. 
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4 Regulator: the agency responsible for ensuring that an organization is 
meeting standards laid down in law. 


5 Political superior: national government in relation to regional govern- 
ment; more broadly the power of states over the private sector. 

6 Mafia: those who may be able to exert force on an organization to act 
differently, often illegally. 

7 Staff: paid or unpaid workers in the organization, or perhaps owner- 
users. 

Another category of stakeholder, not mentioned by Moore et al., is: 

8 Local communities: the whole population of areas where activities are 
undertaken, and their local institutions; likely to include more than just 
the targeted beneficiaries of the organization’s activities. 


Table 8.1 gives examples of how this framework can be used by a UK- 
based international NGO such as Oxfam to determine its stakeholders. A 
national NGO or the country office of an international NGO could be much 
more specific in its use of the same framework. 


TABLE 8.1 
Category of stakeholder Examples for an NGO such as Oxfam 
Owner ; the Board of Trustees 
Financier the public, Department for International 
Development (DfID), European Union 
User “beneficiaries”, ‘clients’, ‘partners’ 
Regulator the (UK) Charity Commission 
Political superior governments (in the UK and in countries 
where operating) 
Mafia guerrilla groups active in particular areas 
Staff paid and volunteer staff 
Local communities local populations, traditional authorities, 
local councils in areas where operating 


This sort of stakeholder analysis can be extended. In this case we could 
add further columns showing the resources each stakeholder brings to the 
organization, what each expects to get out of the organization, and what 
power it has to prevent the organization’s activities continuing success- 
fully. This can be the starting point for understanding the degree to which 
the aims and incentives created by these groups diverge or cohere. It also 
helps us to understand how effective the different stakeholders are in ‘pres- 
surizing’ the organization to perform, and to track how this changes over 
time. 
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8.1.2 PRODUCTION AND REPRODUCTION: THE “GENDER ARCHAEOLOGY’ OF 
ORGANIZATIONS 


This approach is based on the following proposition: 
An organization is both the product of the society in which it is 
situated and an actor in reproducing that society. 
Gender, as an issue in organizations, can be explored fruitfully through 
this approach. Gender is also an interesting vehicle for understanding the 
two-way relationship between an organization and the society in which it 
is embedded. This provides us with an explanation of how one set of inter- 
ests (in this case male interests) are institutionalized. 


In looking at gender and institutions Anne Marie Goetz argues that it is often 
simpler to start by identifying the outcomes, or products, of an organization 
and how these affect men and women differently (Goetz, 1996). This often 
shows up internal reasons why a particular organization produced a specific 
product or outcome. For example, if we see that an agricultural extension 
agency only provides advice to male farmers we might then look at that agency 
to see why it does not recruit women extension agents or why it does not do 
research into crops which women in particular grow. This might in turn lead us 
into looking at the assumptions that organization makes about agriculture and 
who is making those assumptions. This helps us to understand how organiza- 
tions are gendered (i.e. function in ways that reflect gender relations in soci- 
ety) and how they are gendering (i.e. reproducing those relations, which gen- 
erally means reproducing inequalities). 


Goetz suggests that ‘the key to understanding how such outcomes are pro- 
duced is to trace the way institutional structures, practices and agents embody 
and promote gendered interests’. Figure 8.1 illustrates how these interact. 


broader institutional environment (state, market, community) 


the organization 
structure: rules 
and culture 


personal 


: c practices: procedures, 
ideologies 


: fj incentives 
hierachies 


agents 


agents bring personal T= ee a structures and 


ideologies to work in practices create 
ways which affect incentives affecting 
practices and agents 
structures 


FIGURE 8.1 Organization structure, practices and agents (after Goetz, 1996, p. 4) 
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Goetz then goes on to make the point that although structures and practices 
in organizations create incentives and disincentives for staff to behave and 
act in particular ways, those individuals help to create and recreate the 
structure and practice. Understanding the feedback loop between individual 
agents and these structures and practices can give important insights not 
only into how change might be effected in an organization, but also how 
policy directives can be blocked and subverted. Table 8.2 summarizes some 
of the questions that Goetz suggests can help uncover the ‘gender archae- 
ology’ of organizations. 


TABLE 8.2 TOWARDS A GENDER ARCHAEOLO! NIZATIONS 


Institutional history | Who was involved in establishing the organization? 
Who was not? 

How was this done? 

Ideology What are the academic disciplines that animate the 
organization? 

What are the underlying assumptions those 
disciplines tend to make about gender roles? 

Participants Who makes up the organization? 

What is the gender division of labour? 
What is the gender profile of the different levels of 
the organization? 

Space and time Are working hours difficult for women employees? 
Are meetings or social activities held at times 
which might discourage women’s participation? 

Is provision made for career breaks, childcare, etc.? 


Does the organization demand travel away from 
home as part of the job? 
Authority Can women command authority for their approaches 
structures and views in the organization? 
Do women have to take on ‘male attributes’ to be 
heard? 
What is valued in terms of achievements in the 
organization? 
Incentives and On what do performance targets tend to focus; more 
accountability on quantitative rather than qualitative targets? 
To whom and in what direction is accountability 
mainly oriented — upwards, downwards or 
horizontally? 
(After Goetz, 1996) 
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8.1.3 ORGANIZATIONS AND CHANGE 

Organizations are not static unchanging bodies. They are dynamic and 
change in line with changes in their environment and within their parts. In 
a world of growing interdependence and enhanced communications, rapid 
and unpredictable change is becoming more common. It is essential there- 
fore to understand the nature of this change, what provoked it and how it 
occurred. Without this we only have a snapshot of where an organization is 
currently, not a moving picture of how it got there which might also pro- 
vide clues to where it might go. 


It is possible to focus purely on the organization itself, and to treat the 
environment within which it functions as a set of external forces that the 
organization has to cope with. But this gives us a rather limited view of 
how change happens. We can go further in understanding the nature and 
source of change if we treat the organization as an integral part of a wider 
system — interacting with its environment and able to shape and guide the 
forces that produce public action and hence to some extent change the 
nature of change itself. 


This way of understanding change relates to a kind of ‘acupuncture’ ap- 
proach to solving problems, whereby one is not looking necessarily to re- 
move the causes of a problem, which is not always possible, but to alter the 
system through triggering changes that are already embedded in the logic 
of ongoing processes (Morgan, 1986). This means recognizing that most 
change is not linear and predictable but is based on interdependent rela- 
tionships and feedback. Organizations do not behave like machines but are 
dependent on their contexts and the decisions and acts of a multitude of 
individuals. 


The benefit of this sort of analysis is that instead of seeing individuals, 
organizations and public action as separate we start to see them as con- 
nected, overlapping and created by each other. The optimistic side of this 
is seeing the possibility of us as individuals effecting change in our own 
organizations and in doing so changing other institutions (Roche, 1994). 


8.2 TOWARDS SOME GENERIC QUESTIONS 


From the above it seems that there are perhaps four generic areas in which 
we could find out more about organizations. Of course, the way you choose 
to find out about organizations and the focus of the investigation will de- 
pend on your specific purpose. However, the following framework may be 
useful in thinking through investigation. I will be using these categories as 
a means of structuring some of the following sections, which look at dif- 
ferent types of organizational investigation, for different purposes. 


8.2.1 HISTORICAL ANALYSIS 


The sections above on gender archaeology and on change in organizations 
both stress the importance of understanding institutional history and change 
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over time. In order to understand public action at a particular moment, and 
what information is — or is not — available about that action, the historical 
dimension is critical. 


Hans Singer’s review of the lessons of post-war development experience 
provides several examples of how, for example, economic policies have 
developed over time and how this is related to which ‘faction acquires 
ascendancy in and control over important institutions and governments’ 
(Singer, 1989, p. 2). He cites in particular how 40 years ago there was an 
almost fundamentalist belief in planning, balanced growth and import sub- 
stitution whilst in the 1980s export orientation, market power and ‘getting 
the prices right’ provided the basis for a new orthodoxy. 


He also shows how 
Just as generals tend to fight the last-but-one-war, so the develop- 
ment actors as well as the development thinkers seem to base their 
action and thought on experiences of the last-but-one decade or a 
last-but-one phase. 


(Singer, 1989, p. 3; see also Hewitt, 1992) 


This again suggests the importance of relating any analysis of action to the 
political, economic and academic climate not only of the moment of any 
action being taken but also of the preceding period. 


8.2.2 INTERNAL DIVISION 

Organizations are not monolithic entities. The importance of understand- 

ing the fragmented realities of organizations cannot be over-rated. There 

are diverging interests and power relations in any organization which may 
be based upon, amongst other things: 

* informal divisions (e.g. between ‘insiders’ and ‘outsiders’, new employ- 
ees and established employees, people who socialize after work and those 
who do not); 

* functional divisions (e.g. between departments or divisions, between 
professional groupings); 

* hierarchical divisions (e.g. between board members and employees, 
managers and workers, headquarters and other offices, home and ‘over- 
seas’); 

* gender, race or class divisions. 


These divisions will reflect the cultural, political and economic context in 
which an organization is located as well as being reproduced within the 
organization itself. It is therefore important to understand the divisions. 
One might want to exploit them (for example by a lobbying initiative 
seeking to influence an organization to adopt a particular position). It might 
be important to overcome divisions in order to improve performance. 
Or one might simply need to make sure that a particular project does not 
exacerbate existing tensions and cause conflict. 


179 


THINKING WITH PEOPLE AND ORGANIZATIONS 


8.2.3 PERFORMANCE, TRANSPARENCY AND ACCOUNTABILITY 

As Goetz (1996) and Moore et al. (1994) suggest, the relationship between 
the values and culture of an organization and the incentives placed upon it 
by its various stakeholders, is important in determining performance and 
impact. Understanding how the identity, values, rules and norms of an or- 
ganization have been constructed is therefore vital. It is equally important 
to analyse to whom the organization is accountable, and how multiple ac- 
countability is managed. 


8.2.4 LEARNING SYSTEMS, CHANGE AND ADAPTATION 

Understanding about the internal divisions in an organization, its perform- 
ance and its accountability mechanisms can give us a good picture of cur- 
rent reality. However, we will often want to know about an organization’s 
potential for developing in the future. I have already (in Section 8.1.3) 
highlighted the need to understand the overall dynamics of change, of which 
the organization is only a part. This raises questions about the organiza- 
tion’s ability to learn from experience — in adapting to and changing its 
environment. 


8.3 BACKGROUND INVESTIGATION OF AN ORGANIZATION 


How exactly you investigate an organization will depend on your particu- 
lar purpose. The four areas in Section 8.2 will help you to formulate some 
more precise questions, but first you will need to undertake some back- 
ground analysis, using literature and documents which are readily avail- 
able, noting the organization’s public behaviour and reputation, and per- 
haps asking some preliminary questions both inside and outside the or- 
ganization. 


8.3.1 WHAT AN ORGANIZATION SAYS ABOUT ITSELF 

The first place to start is to get hold of what an organization says about 
itself. Annual reports and accounts, public relations material, strategic plans, 
publications, and recruitment material are all useful but need to be read 
critically (see Bridget O’ Laughlin’s Chapter 5). Some of the questions you 
might like to ask yourself during your reading of such material are in- 
cluded in Box 8.1. 


It should be remembered that this material represents the public face of an 
organization and even when that includes quantitative information, such as 
accounting figures, this needs to be interpreted. As Hopwood and Miller 
(1996) make clear in their book Accounting as Social and Institutional 
Practice ‘accounting is no longer to be regarded as a neutral device that 
merely documents and reports ‘the facts’ of economic activity’. Accounts 
too are based on arbitrary classifications, political interests and compro- 
mises and are the product of the institutional dynamics noted earlier (see 
Chapter 12 for more on interpreting institutional accounts). However hav- 
ing this information to hand is a useful first step in the processes of finding 
out more and contrasting the public image of an organization with its more 
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BOX 8.1 WHAT AN ORGANIZATION SAYS ABOUT ITSELF 


History 

¢ When was the organization established, and for what purpose? 
* In whose interest was it established? 

* What major changes have occurred over time? 


Identity, values and strategy 

* What sort of identity and values does the organization convey through its 
material? 

© What is the organization’s legal status? 

* Is the organization’s vision, and strategy for achieving that vision, clear? 

© What statements does the organization make about its attitude to equal 
opportunities, gender and racial equality, staff development? 

* How many staff are there? Who makes up the organization in terms of 
gender, race, age? What skills, disciplines and abilities are there? 

* Who are the Board of Trustees, directors, or ministers who ultimately are 
legally responsible for the proper running of the organization? What in- 
terests do they represent? 

Systems and structure 

* How is the organization structured? How is that structure presented or 
described: as a hierarchy. on a functional basis, on a geographical basis? 

* What sort of formal decision-making processes exist? Are there particular 
committee structures, annual meetings, general assemblies, board meet- 
ings? How often do they occur? 

Finance 

* How is the organization funded and by whom? 

© How is the money spent? How much is devoted to different functions 
especially research and development, monitoring and evaluation, training 
and administrative costs? 

* What is the income and expenditure and by how much does one exceed 
the other? What is done with any ‘profit’? What ‘reserves’ does the or- 
ganization have? How long could it survive if it received no further in- 
come? What assets does the organization have? 

Performance 

* What does the organization say about its achievements and performance 
in terms of effectiveness, efficiency and impact? 

© Who is assessing that performance and on what basis? 

© To what extent are problems, weaknesses and ideas to overcome them 
evident in this material? 

© Are diverging views of performance made transparent? 

Relationships 

© Who are the major stakeholders that can influence the organization, posi- 
tively and negatively, to adopt different policies? 

* What alliances does the organization have with other actors? 
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‘private’ one. This material is probably most useful in answering some of 
the questions about the history of an organization and what it considers its 
most important achievements to communicate are, and may provide some 
indication of corporate plans for the future. It is usually less revealing about 
internal differences, its accountability to less vocal stakeholders, and any 
errors made, failures or problem areas. 


8.3.2 WHAT STAFF AND EX-STAFF SAY ABOUT THE ORGANIZATION 

Meeting, observing, and listening to individuals from organizations, in 
public fora, at meetings, and in the normal course of professional interac- 
tion, can tell you a lot — particularly about the culture of the organization 
they work for or used to work for. The questions in Box 8.2 are a useful 
checklist for this sort of interaction. (And particularly useful if you want to 
make the most of any opportunities to talk to staff in an organization you 
are or will be investigating!) 


BOX 8.2 WHAT STAFF SAY ABOUT THE ORGANIZATION 


© Do staff tend to say the same thing or different things? Is this to do 
with their role or not? 

¢ Is there a consistency across the organization? About what? 

¢ How critical of the organization are they? How open to ideas from 
others? 

© What elements of their organization do they emphasize (i.e. achieve- 
ments, failures, structural issues, staffing issues)? 

* What change would they like to see in the organization? How do 


they think this might be achieved? 


8.3.3 WHAT OTHERS SAY ABOUT THE ORGANIZATION 

As we have noted above any organization is influenced and pressurized by 
a variety of interests. What are others saying? This might include peers, 
adversaries, pressure groups, the press, academics, users or other interest 
groups. Getting this information is likely to require a mixture of searching 
for articles and other ‘grey material’ that include something on the organi- 
zation you are concerned with, and opportunistic conversations with indi- 
viduals from the various groups whose opinions you want to collect. Box 
8.3 gives some questions to guide this part of your investigation. 


The cross-checking of information between these sources can help to verify 
or discount facts, opinions and rumours. It can also be the means by which 
more precise questions can be formulated to explore in more detail. 


Exploring and putting together different interpretations from staff and oth- 
ers is a key part of exploring the potential internal divisions within an or- 
ganization. In addition as Joseph Hanlon in Chapter 3 notes ‘finding the 
woman who knows’ can save a lot of time and energy. 
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BOX 8.3 WHAT OTHERS SAY ABOUT THE ORGANIZATION 


* Who are these other informants? What are their particular inter- 
ests? Who do they represent? Where or from whom do they get 
their information? 

* Is there some consistency in their views? What do they empha- 
size? 

* How would they rate the organization and what do they think is 
the general view of it according to other actors? 

* What change would they like to see in the organization for their 
purposes? How do they think this might be achieved? 


8.4 ORGANIZATIONAL ASSESSMENT AND ORGANIZATIONAL 
CHANGE 

In this section I will concentrate on how to ‘find out fast’ about organ- 
izations and public action when working with people in those organiza- 
tions, with a view to promoting organizational change. It draws heavily 
on my own experience of facilitating and being involved in participatory 
exercises with NGOs. Johnson and Mayoux point out in Chapter 7 
the inherent contradictions of any participatory investigation. These 
apply equally to participatory investigations in organizations. However, 
organizational assessments need not overtly be labelled ‘participatory’ and 
may even be conducted from outside with a view to catalysing change 
within the organization. Much of what follows would remain relevant in 
any case. 


Helping an organization reflect on what it does, how it does it and what 
its strengths and weaknesses are is not only a way of analysing its in- 
volvement in public action but is also a means of improving its ability to 
do so. Indeed as suggested above this capacity for self-reflection is a key 
element in an organization’s ability to change in line with its external en- 
vironment. 


The rest of this section follows the four generic areas laid out in Section 
8.2 above. 


8.4.1 HISTORICAL ANALYSIS: TIME LINES AND MATRICES 

Exercises in constructing the history of an organization are important for 
participants who often have varied if not conflicting understandings of an 
organization’s origins and background. Confronting these different inter- 
pretations, confirming facts and identifying gaps in the historical under- 
standing of staff provides an important foundation for constructing a joint 
understanding of how how an organization got to where it is. For these 
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Primary policy thrust Programmes 
1975 Rehabilitation Mali — Gao Region 
Southern Sudan 
1979 Institution building Tanzania (Kiu) 


Southern Sudan (Amadi) 


1981 Production systems Mali — Timbuktu Region 


Sudan — Qal-En-Nahal, Fort 
Sudan 


Uganda Gulu, Fakwach 
Benin — Materi 
Burkina — Dori 


Somalia — Sablaale 


1985-8 Reinforcing local structures Mali — Gourma 
Somalia (expansion) 
Sudan Red Sea Hills 
Angola (Adra) 
Uganda (redesign) 
Guinea 
Tanzania 
Sudan — Juba 
Rwanda (Aramet) 

1988 Women in development (WID) Mali (redesign) 
Rwanda 
Mozambique 

1990 Gender Angola 
Tanzania 

1992 Burundi 
Niger 
Namibia 


FIGURE 8.2 Time line of ACORD’s policy development up to 1994 
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reasons, the time line of the international NGO consortium ACORD 
(Agency for Cooperation and Research in Development) (see Figure 8.2) 
was constructed by its headquarters staff during a ‘Participatory Organiza- 
tional Appraisal’ of the organization facilitated by myself and Mick Howes 
of the Institute of Development Studies at the University of Sussex (Howes 
and Roche, 1995). Although the primary function of such an exercise is to 
generate agreed understanding for the participants, the information gained 
is also useful for the facilitator as an outside investigator. 


What is interesting about the process of constructing the time line was the 
way it revealed the origins and subsequent policy development of the or- 
ganization. ACORD was set up in 1973 in response to the Sahel drought 
and the civil war in southern Sudan. Later it was influenced by the ‘cli- 
mate’ of events and development thinking, yet the programmes that were 
developed in different policy epochs still carried the imprint of the organi- 
zation’s origins. For instance those programmes established when one of 
the primary policy aims was production-oriented (1981) had great diffi- 
culty in adapting to some of the demands made upon them in later years, 
when gender concerns (1988) and local structure development (1985-88) 
became more central. 


If staff are able to think through a time line in this way, it can lead not only 
to a better understanding of why things are as they are. It also helps them 
work out how their current decisions may affect future possibilities and 
choices. 


Below are two examples of time matrices (Figure 8.3 and 8.4) which at- 
tempt to synthesize data in order to understand the linkage between the 
past and the present. 


Figures 8.3 and 8.4 were used during management induction courses in 
Oxfam in order to: 


1 present to participants the evolution of Oxfam since its inception along 
several different axes (staff numbers, income, programme development, 
campaign work, management culture). Figure 8.3 indicates how the 
changing size and management culture of the organization paralleled a 
growing campaigns focus and an evolving programme development 
strategy. Periodic involvement in major disasters and conflicts remained 
a constant feature and a key factor in organizational growth. 

2 situate Oxfam in the wider context of the British NGO sector and de- 
velopment policy in general. Figure 8.4 shows how the NGO develop- 
ment agenda has gradually broadened to include trade and advocacy 
issues whilst also reacting to (and against) the priority concerns of do- 
nors. 
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2 Save the Children 
Chrieban Ald donor concerns 
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FIGURE 8.4 Evolution of British NGO sector; WID is ‘women in development’ 


The advantage of laying out history, or constructing such time matrices 
with staff in this way, is that it allows insights into the relationship between 
different elements occurring in parallel as well as showing historical shifts 
over time. For new staff joining Oxfam it can help them to understand the 
British NGO scene and where and how Oxfam fits in. 


8.4.2 UNDERSTANDING DIFFERENCE AND INTERNAL DIVISION 

One important element that emerged from the appraisal mentioned above 
at ACORD was the distinction people made between the official and unof- 
ficial hierarchy in the organization, the importance of personal communi- 
cation networks and how some felt that those already in more powerful 
informal networks did not see the need for more formal communication 
channels. In this case encouraging people to map their perceptions of per- 
ceived lines of responsibility and decision-making and to draw their per- 
sonal communication networks proved a good means of exploring issues 
of power and difference and providing the basis for further discussion of 
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the issue. In other cases bringing out such potential conflict may be dan- 
gerous, inappropriate or raise expectations which cannot be met. 


In some organizations this sort of process may require working separately 
with particular groups, for example men and women, managers and non- 
managers, in order for them to feel free to express themselves. 


It is likely that in analysing organizations and public action different opin- 
ions will be given about the same issue. Dealing with this discordant infor- 
mation is an important element of analysis. It is important to capture this 
diversity rather than diluting it through aggregation. 


One important way of capturing this diversity is to present differences of 
opinion in people’s own words. This allows the reader to interpret what is 
going on for themselves (rather than be given your interpretation of it), and 
read between the lines. The following provides an example of this. It comes 
from research by Oxfam into the way the Zimbabwe Government intro- 
duced user fees for health care and how regulations regarding exemption 
were implemented. It shows how talking to particular groups, in this case 
users, can give conflicting views of how an organization works in practice. 


The attitudes of staff to claimants can have a significant effect on 
access to welfare payments. Roda Neune is a married woman with 
three children. She lives in Sigangatsha, South Matebeleland, while 
her husband works in South Africa. She earns a little money from 
selling beer, and qualifies for an exemption: 


‘I can take a letter to the clinic from my village development 
worker. The worker will give me the letter when I need it, after 
asking questions about my husband and my work. It doesn’t cost 
anything.’ 

But Sophia Mutare’s experience of the system was less positive: 


‘I visited the hospital and said that I had no money to pay, so they 
asked questions about my husband, looked at my clothes, and then 
asked other people what they knew about my family. It’s bad that 
they decide like this but you cannot do anything.’ 

(Lennock, 1994, pp. 25-6) 


Another way of capturing diversity of opinion is to ask different groups to 
rate their organization according to an agreed scale and then compare these 
scores in order to discover key areas of divergence and agreement. Nor- 
man Uphoff (1987), in his field methodology for participatory self-evalua- 
tion, suggests a simple four point scoring system where: 


3 represents a most satisfactory situation with little room for improve- 
ment; 


2 represents a satisfactory situation but with some room for improvement, 


1 represents an unsatisfactory situation with considerable room for im- 
provement; and 


188 


8 ORGANIZATIONAL ASSESSMENT AND INSTITUTIONAL FOOTPRINTS 


0 represents a very unsatisfactory situation with very great room for im- 
provement. 


If we see in an organization that senior managers for example are 
constantly rating women’s participation in decision-making at 3 and ju- 
nior females are rating this at 1 then it is clear that important differences 
exist. 


8.4.3 ORGANIZATIONAL DIAGNOSIS; PERFORMANCE, TRANSPARENCY AND 
ACCOUNTABILITY 

Another important element in any self-assessment is an honest appraisal of 
an organization’s identity, values, and strengths and weaknesses. There are 
many suggestions, by different authors, about what this should include (see 
Fowler, 1995, Campbell, 1992 for the private sector and NGOs; see Uphoff, 
1987 for membership organizations; CETI, 1992 for small businesses; 
Zadek, 1996 for value-driven organizations; and Drucker, 1990 for non- 
profit organizations). Box 8.4 is a synthesis of some of the major elements 
put forward by these authors. However it can be better to let staff construct 
their own categories for what they deem to be the most important elements 
on which an organizational diagnosis should focus. 


The importance of this sort of participatory diagnosis is that it leaves the 
categorization and classification of organizational elements in the control 
of staff rather than imposing upon them a necessarily general theoretical 
framework. The result is therefore more likely to reflect their priorities and 
concerns. And although the result may miss out some of the elements in 
Box 8.4, what is missed out is in itself an indication of where an organiza- 
tion is at in terms of its own self-analysis. Furthermore this sort of analysis 
can lead to other elements being included which are often not referred to in 
the generalized checklist of Box 8.4. These might include issues particu- 
larly related to local circumstances and culture. Tracking what self-appraisal 
processes focus upon is therefore an important means of assessing organi- 
zational development and change. 


In addition to participatory self-assessment, it is important to analyse the 
relationship between an organization’s stakeholders and its performance. 
This is crucial in order to understand who is assessing performance and on 
the basis of what criteria. 


In analysing the design of policy and the impact of public action it is once 
again important to capture diverse opinions, to seek out those voices that 
are not usually heard, to juxtapose those views against the orthodoxy and 
to enrich public debate by doing so. Box 8.5 lists critical features that are 
useful in assessing the degree of different groups’ involvement in the policy- 
making process. This type of methodology can be used in many contexts. 
This example is adapted from a study of the Bangladesh Flood Action Plan 
commissioned by Oxfam and other NGOs in Bangladesh (Adnan et al., 
1992). 
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BOX 8.4 ORGANIZATIONAL SELF-ASSESSMENT CHECKLIST 


Identity and values 

To what degree are the organization’s identity and values explicit and 
agreed? 

Vision and strategy 

Does the organization have a clear vision of what it wants to achieve 
and a strategy to achieve it? 

Resources 

Does the organization have the financial, human and technical resources 
it needs to achieve its vision? How effectively does it use these re- 
sources? Where do these resources come from? 

Structure systems and procedures 

Does the organization have the systems and procedures necessary to 
use the resources at its disposal to achieve its vision and strategy? 
Organizational culture 


Is the culture of the organization consistent with its values and its 
vision? How open and transparent is it both internally and externally? 
How are staff motivated and rewarded? How does the organization 
deal with difference and conflict? 


Control and accountability 

Who owns and controls the organization? Who is the organization 
designed to serve and how does the organization know if it is succeed- 
ing in doing so? 

Programmes and services 

What does the organization do? Is what it does consistent with what it 
wants to achieve and its values? 

Performance and results 

How well does it perform in relation to its own vision and strategy 
and in relation to other organizations with similar goals? 

Learning and change 


How well does the organization learn from its experience and that of 
others and adapt in the light of that learning? How well does the or- 
ganization cope with change? 


Leadership, management and decision-making 


How effective is the leadership and management of the organization? 
How consistent is it with the values of the organization and what it 
wishes to achieve? 
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Cross-cultural issues 
External linkages and relations. 


To what degree is the organization externally oriented and to what 
degree does it enter into strategic collaboration with others? 


(Adapted from Campbell's organizational effectiveness questions (1992), Fowler’s 
Community Organization checklist (1988) and his work on participatory self- 
assessment of NGO capacity (1995), Drucker’s questions for organizational self- 
assessment (1993), and dimensions for benchmarking coming out of a NOVIB 
partners meeting in 1996.) 


BOX 8.5 CRITICAL FEATURES IN ASSESSING INVOLVEMENT 
IN POLICY DEVELOPMENT 


1 Transparency 


To whom are the different stages of policy development visible, 
including the decision-making and implementation processes? 


2. Access to information 
Who has access to adequate and timely information on policy for- 
mulation, design and monitoring? 

3 Accountability 
To what degree are the agencies involved in policy development 
and implementation procedurally and periodically answerable to 
different groups in the impact areas, as well as the citizens of the 
country in general? 

4 Meaningful choice 
Who can participate in policy development in a voluntary manner 
without being compelled, constrained or otherwise left with no 
other choice? 


5S Comprehensive 
Who has been consulted from the very outset in defining the na- 
ture of the problem, as contrasted to those consulted at other stages? 
6 Non-alienation 


Have people participated in a way that they do not feel distanced 
and alienated from policy actors, the implementation process and 
the eventual outcomes? 


(Adapted from Adnan et al., 1992) 
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8.4.4 LEARNING AND ADAPTABILITY 

One of the quickest ways to gain an understanding of organizational change 
is to investigate how organizations learn and adapt. Learning audits 
and self-assessment questionnaires are now becoming increasingly com- 
mon tools to involve staff in analysing existing processes. Part of this is 
in recognition that the emphasis may need to shift from the more formal 
learning systems, for example monitoring and evaluation and the usual 
reporting systems, to more nimble, creative and less formal learning and 
exchange. 


Box 8.6 suggests some of the questions that might be explored with 
staff. 


Recent exercises of this type undertaken in Oxfam and other NGOs have 
revealed some common constraints on organizational learning which are 
perhaps not confined to NGOs, for example: learning is seen as an ‘over- 
head’ cost; incentives for learning and admitting error are weak; systems 
for recording and disseminating information are cumbersome and ineffi- 


BOX 8.6 QUESTIONS RELATED TO ORGANIZATIONAL LEARNING 


* Who learns in the organization and how? 

© What kind of learning is rewarded? 

* To what degree are errors admitted and analysed? 

* What forms of knowledge are legitimated and how? 
¢ What constraints are there to learning? 

* How does information flow in the organization? 


¢ Howis institutional memory constructed, how accessible is it, and 
to whom? 


© What changes occur through self-learning as opposed to other in- 
fluences? 


* How does the organization react to learning which challenges its 
assumptions? 

* Is the organization better placed now to anticipate change in its 
environment and adapt accordingly? 


* What changes are being made to the organization’s learning sys- 
tems? 


(Adapted from Edwards, 1996, Howes and Roche, 1995) 
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cient; the need for success stories is high (Edwards, 1996, Howes and Roche, 
1995). 


These studies also reveal that when analysing non-profit organizations in- 
cluding government agencies we need to bear in mind the differences be- 
tween them and the profit-making organizations on which much of the 
‘learning organization’ literature is based. These differences stem partly 
from the fact that NGOs and governments do not face the discipline of 
market forces in the same way as private companies which can be wiped 
out if they do not learn and adapt ‘efficiently’. This difference is com- 
pounded for NGOs by the fact that the ultimate ‘customers’ of their serv- 
ices do not guarantee the survival of their organizations. 


However, if as I suggested in Section 8.4.3 finding a single framework for 
analysing and presenting data on performance and accountability is diffi- 
cult, then to do so for learning and adaptability is even more problematic. 


One possibility is to explore and present different levels of learning in any 
organization — individual, team, departmental and organizational — on the 
basis of questions suggested in Box 8.6. Another is to look at functional 
divisions of learning, such as experiential learning, project-based learning, 
policy-related learning, advocacy-based learning and scientific research 
and visionary thinking (Edwards, 1996). 


However we also have to be aware of the links between any organization 
and those external to it but who nevertheless influence its development. 
Studies such as that by Judith Tendler and Sara Freedheim in northeast 
Brazil demonstrate how important this is. In their study they show how the 
high performance of government health workers is partially due to the trust 
and respect that they had from their clients in the community (of which 
they were a part), as well as their relative autonomy to respond to them in 
ways they felt appropriate (Tendler and Freedheim, 1994). This throws up 
interesting issues relating to current western policies which continue to be 
based on the mistrust of public sector workers and their supposed ‘rent- 
seeking’ behaviour. 


From the point of view of ‘finding out fast’, monitoring what happens at 
the interface of an organization with the outside world, in terms of its rela- 
tionships with its primary stakeholders and how they shift over time, gives 
an interesting insight not only into how receptive an organization is to its 
stakeholders, but also into how well it learns and adapts as a result of that 
experience. 


Some possible measures of how well an organization is learning and how 
adaptable it is are set out in Table 8.3. They have been developed by the 
NGO ACORD building on the ideas of Tom Peters (1992). 
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IBLE MEASURES OF LEARNING AN‘ 


Listening better 
1 To improve quality 


2 Making contact 


3 Listening 
management 
Efficiency of learning 


4 Greater 
responsiveness 


5 Constant innovation 


6 Keeping a degree 
of specificity 

7 Learning from 
other agencies 


8 Management 
support to innovation 


9 What has changed 
as a result of learning 


Indicators 


Is the organization open to continuous quality control by 
those it seeks to benefit and open to ideas from others? Do 
measures of quality exist? 


How and how often does the organization talk/listen to its 
clients and funders? 


What is the percentage of time spent with ‘front-line’ staff on 
the ground and key decision makers in other organizations? 


How long does it take for the organization to respond to 
or deal with newly identified needs or new situations? 


How involved is the organization in action research, pilot 
initiatives, etc. and what lessons are being learnt from these? 


What differentiates the organization from other agencies? 


How many ideas have been gained through discussion with 
or observation of other agencies? 


Is the analysis of interesting past failures encouraged? 
What is the degree of acceptance of failure? Is the 
organization learning from past failure? Does it admit past 
failure? Does it undertake studies to understand errors? 


How much does the organization change from year to year? 
What adaptation or modification in procedures, structure, 
staffing, methodology, activities, objectives, indicators has 
occurred? Do reports reflect these changes? 


Organizing to react to rapid change 


10 Decentralized 
marketing and 
fund-raising 


11 Word of mouth 
marketing 
and fund-raising 


12 Developing multi- 
functional teams 
13 Training 


14 A changed middle 
management role 


15 De-bureaucratization 
16 ‘Front-line’ focus 


(After Peters, 1992) 


How many funder visits to the organization, how many 
staff visits to funders? 


How much time is spent selling ideas and programmes 
face-to-face with funders? 


Percentage of people/problems solved in multi-layered, 
multi-disciplinary team configurations 

Hours/pounds devoted to skill upgrading of ‘front-line’ staff 
as opposed to other staff 


Number of acts of boundary bashing, degree of insistence 
on flexible bottom-up quality control measures, creation of 
supportive learning climate 

Number of unnecessary procedures, regulations, meetings, 
committees renounced per month 


Number of ‘front-line’ as opposed to 
middle-management staff at meetings 
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The logic behind most of the measures in Table 8.3 relates to: 


* How closely and how well an organization listens to those it is trying to 
support and to those who fund it. If organizations are constantly listen- 
ing to those they support then they should be able to adapt more quickly 
and if they are listening to donors/supporters then potential misunder- 
standings and delays can be more rapidly identified. (Items 1-3) 


* How quickly and efficiently the organization learns from listening and 
modifies its support accordingly. Translating listening into action, being 
open to change, constantly testing new ideas, and learning from others 
and past mistakes are all essential facets of being able to respond to 
turbulence and rapid change. (Items 4-9) 


* How well the organization structures itself to promote learning and in- 
novation. In order to react more effectively organizations need to struc- 
ture themselves in a way that allows staff to take decisions where appro- 
priate, that regroups different interest groups to solve problems on the 
spot, that makes line-management of horizontal learning such that it does 
not put bureaucratic hurdles in the way, and that insists on the primary 
focus being the ‘front-line’. (Items 10-16) 


8.5 INSTITUTIONAL FOOTPRINTS AND PUBLIC ACTION 

On occasion the aim of an investigation is to inform or influence public 
action on the part of your own or another organization, and you need to 
examine public action produced by yet another organization as part of that 
investigation. Then as a researcher you will need to look at the ‘footprints’ 
left behind in order to learn about the organization that made them. You 
may also find out about that organization more directly, but the methods 
used would then be interviews and interpretation of written documents with 
no specific use of participative techniques. 


Investigating another organization from outside in this way can pose a 
number of challenges. It is often hard to decipher the power centres of 
organizations and for the investigator this means that it is difficult to judge 
how to conduct investigations and interpret reports. Some of the techniques 
and frameworks described above (e.g. the checklists in Boxes 8.4, 8.5 and 
8.6), could be helpful in this type of investigation although the technique 
and approach will clearly depend on the aim of the investigation. 


A recent study commissioned by the European Union and relevant Polish 
ministries was designed to examine the funding of science and technology 
policy in Poland. The objective here was to understand how the system 
was operating, what impact it was having on the science and technology 
community and whether or not the policy needed reform. What became 
apparent very early on in the investigations is that any attempt to under- 
stand the policy without extensive examination of the principal ministry 
involved would be grossly misleading. Historical analysis and understand- 
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ing the divisions within the ministry were particularly important. For 
example, while policy statements indicated a concern for technology fund- 
ing as well as more basic science, in practice the mechanisms for funding 
more basic work rather than technological applications were much more 
entrenched. In interviews it became clear that this situation was not the 
product of a policy shift, so much as the unintended consequence of insti- 
tutional evolution and cultural bias. The ministry was very keen to protect 
its autonomy and to protect scientists from the type of interference that 
they had experienced during the previous era of central planning. This hin- 
dered the establishment of a peer review system and construction of a net- 
work of institutions which could have adequately addressed technologi- 
cally oriented development as well as basic science. In this case, an under- 
standing of explicit policy goals was complemented by an appreciation of 
cultural bias, historical factors and different interests within the Ministry. 


Consultants carrying out the study based themselves in the Ministry and 
talked to a wide range of ministry officials. Interviews with other 
stakeholders not only gave important indications about the results of policy, 
but also provided consultants with ‘pointers’ as to the different power cen- 
tres within the ministry. The consultants gained an understanding of how 
different stakeholders experienced policy, who they considered important, 
who they talked to or did not talk to. In the Ministry interviews helped 
investigators construct a ‘map’ of the Ministry itself. 


On other occasions ‘institutional footprints’ may be all you have to go on. 
This is when organizations do not allow you in to analyse their organiza- 
tional dynamics and the effects of their actions, but you still need to find 
out about them in order better to understand a situation. In such cases you 
may have no direct access into the organizations. What else can be done 
‘from the outside’ over and above looking at their published and grey ma- 
terial? How can you ‘find out fast’ about public action in this situation in a 
way that will still be credible? The following example from Zimbabwe 
tries to illustrate how this might be done and how understanding of organi- 
zational dynamics and the context in which they operate can be helpful in 
arguing for policy shifts. It comes from the same Oxfam investigation into 
the effect of introducing user-fees for health care in Zimbabwe quoted in 
Section 8.4.2 


Oxfam in Zimbabwe in the early 1990s became very concerned about the 
effect on low-income groups of the Zimbabwe Government and World Bank 
policy on cost-recovery through the raising of user-fees for health care. 
The Government’s framework for Economic Reform (1991-5), drawn up 
in collaboration with World Bank and the International Monetary Fund 
(IMF), aimed to increase revenues from user fees from Z$15 million in 
1989/90 to Z$60 million by the end of 1995. In 1991 households earning 
cash incomes of less than Z$150 a month were entitled to exemption from 
paying fees. This was increased to Z$400 a month in 1993. 
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This policy, pushed by the World Bank and willingly embraced by the 
Zimbabwe Government, can mainly be attributed to a caucus of US-trained 
health economists keen on pursuing models of health care privatization. 
The policy was based on the belief that price-based systems of rationing 
are the most efficient way to cope with growing demands for health care. 
People already paid for a proportion of their health care. The assumption 
was that they were willing to pay for health care and were therefore able to 
pay. Those charged with the implementation of the policy, including World 
Bank staff, were often less convinced as to the likely practical outcomes. 


In order to investigate the potential differences between theory and prac- 
tice Oxfam hired a consultant in 1993 to undertake a relatively rapid piece 
of research which would pull together existing material as well as comple- 
ment this with primary research of a number of health service users. 


The researcher pulled together secondary data on the effects of user fees 
on low-income groups as well as statistics regarding attendance at ante- 
natal clinics and trends in maternal and prenatal mortality — using Oxfam’s 
existing network of partners and contacts as well as UNICEF, the Ministry 
of Public Service, Labour and Social Welfare, the Ministry of Health and 
Child Welfare, and Save the Children studies. In addition 16 interviews 
were conducted with rural women over a two week period in August 1993 
in Manicaland, Masvingo, South Matebeleland, North Matebeleland, and 
North and East Matebeleland. 


The resultant publication drew upon the secondary data but illustrated the 
human consequences of the policy with more qualitative data collected 
through the semi-structured interviews (including the examples quoted in 
Section 8.4.2; Lennock, 1994). 


The findings of the report confirmed the growing body of research which 
indicated that: 


* user-fees have the effect of pricing health services beyond the means of 
those most in need; 


* exemption systems were too complex and too arbitrary and therefore 
did not effectively protect the access of poor people to health services; 


* as in many poor countries where a lot of users may be exempt, the costs 
of administering the system are greater than the revenue gained; and 


* for poor people, user-fees set at even modest levels undermine access to 
basic health care. 


In addition the report concluded that: 


* asthe revenue generated by user-fees is transferred to the general budget 
it is therefore not necessarily used on improving the health system; 

* other revenue earning options which would be less damaging to the poor 
such as a land tax or higher corporate taxes were rejected for political 
reasons; 
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* that the government's claim that user-fees were needed to channel pa- 
tients to the most appropriate point of entry to the health care system 
ignored the fact that patients often sought access to higher-level facili- 
ties because of the collapse of primary health care provision. 


On the basis of the above evidence the report recommended that ‘having 
initially endorsed the case for user-fees, the Bank has a responsibility to 
use its influence to press for their withdrawal. It could do so most effec- 
tively by demanding the phased withdrawal of user-fees over the lifetime 
of the adjustment programme’ . 


The report caused a furore in Zimbabwe with Oxfam being accused amongst 
other things of having a political agenda by deliberately releasing the re- 
port during the run-up to the 1995 election; wanting to undermine the Zim- 
babwean state by undermining their position in negotiations with donors; 
wasting money destined for charitable purposes on spurious research and 
making up stories. Oxfam at one stage was threatened with expulsion from 
Zimbabwe. However, thanks to the experience and political expertise of 
the Oxfam representative in Harare, as well as backing from Oxfam head- 
quarters in Oxford, instead of backing down Oxfam stood by the findings 
of the report which ironically were borne out by the government’s own 
research. 


During this period and despite getting into processes that were unforeseen, 
Oxfam was able to point to the research which, despite its time scale, was 
rigorous enough to withstand much of the criticism as well as forming the 
basis for a detailed rebuttal of the accusations. Richard Jolly, the Deputy 

Director of UNICEF, stated that he was ‘consistently impressed by 

* its careful analytical tone and style; 

* its professionalism; 

* the way its argument was based overwhelmingly on national data 
and general principles, with argument from particular districts, hospit- 
als or individuals used only to illustrate more general points with specif- 
ics (in a way which matches closely UNICEF's own methods of advo- 
cacy); 

* its relevant references, 11 about Zimbabwe.’ 


He went on to say that ‘the evidence [is] well marshalled and convincing 
... I see no reason for treating the study, though brief, as anything but a 
first rate piece of work ...’ 


In the end there was a relaxation by the World Bank of its policy on user- 
fees and the Zimbabwean Ministry of Health conceded that cost-recovery 
had resulted in adverse outcomes, especially for the rural poor. It suspended 
cost-recovery for rural health clinics in 1995, except for a nominal regis- 
tration fee. In a post-drought phase this relaxation of policy was particu- 
larly important at a time when people were seeking to rebuild their asset 
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base. The degree to which this is attributable to Oxfam and the report are 
debatable. Clearly many factors combined to produce this outcome and it 
will never be clear whether such policy change would have happened any- 
way. 


Although there were some particular circumstances that together with the 
report contributed to the policy change, and although that change was tem- 
porary, this example does illustrate how public action can be analysed in a 
way that, along with other factors, leads to changes in policy. 


Some of the success factors one can distil from this example include: 


* Oxfam’s understanding of differences of opinion within the World Bank 
indicated where there was room for change; 


* the research explored a dimension where a professed policy of poverty- 
reduction by the World Bank was contradicted by the practice of intro- 
ducing user-fees; 

* the personal involvement of a senior World Bank official in visits to a 
mission hospital to see for himself the results of the introduction of user- 
fees; 

* the ability of individuals in Oxfam to seize opportunities which unex- 
pectedly presented themselves when the report provoked a wider debate 
in Zimbabwe, i.e. making the most of luck and coincidence; 

* the support received from some partners with whom Oxfam had built up 
relationships of trust and mutual respect over many years, although most 
did not want to go public on the issue, as well as support from the Zim- 
babwe Congress of Trade Unions and from members of the public. 


Overall this example illustrates again the importance of understanding the 
historical and organizational dynamics behind any policy, in this case the 
introduction of user fees. Combining this with detailed research on the 
performance and implementation of the policy and a good understanding 
of the kinds of incentives for change which had effected and would affect 
the World Bank and the Zimbabwe Government was critical. As Mayer 
points out in Chapter 13 good quality data on their own are not enough. 


In addition the process involved a constant reassessment of Oxfam’s rela- 
tionship with the Zimbabwe Government and the World Bank, at times 
aligning with individuals within these organizations and at other times con- 
fronting them. Being capable of adapting in the light of changing circum- 
stances was a necessary ability. 


8.6 CONCLUSION 

At the heart of many of the ways of analysing public action presented above 
is an implicit understanding that this analysis is used to help promote change. 
This can be change within organizations through a better understanding of 
what they do. Alternatively it can be broader public action, either change 
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in the relationship between organizations through a better understanding 
of how they relate, or change in the policy environment through a better 
understanding of how it is made up and how it might be altered. 


This chapter has attempted to explore different ways of analysing public 
action through developing our understanding of where organizations are 
coming from, who makes them up, how they perform and who the decision 
makers are, and how they learn and adapt. Most of the examples that have 
been included come from my own experience of work within the NGO 
sector. It is therefore a chapter that necessarily comes from a certain per- 
spective; one that recognizes the crucial role of citizens in shaping, and 
being shaped by, public action. Those with more experience in the state 
sector might have looked at the issue from a different perspective. 


However although the chapter has an activist and NGO perspective I feel 
that some of the ways of looking at the issue of organizational assessment 
and institutional footprints could be equally valid for someone working in 
local government or in private sector service delivery. Although there are 
key differences between public and private sector organizations there are 
also similarities notably in terms of the ways bureaucracies tend to func- 
tion and recreate themselves, and in the need for organizational learning 
and informed change. 
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DINA ABBOTT 


In 1989 I carried out a study on how poor women in Bombay survive by 
making a living from home-based activities. The gathering of evidence 
took a few months and provided the empirical basis for my PhD. The analy- 
sis and writing up, however, took some five years from start to finish. The 
topic was not simply one of academic interest. Poverty is of basic concern 
to development policy. Although this particular study was not directly con- 
nected to policy formulation, it, and the access to poor women it required, 
was only possible because of initial support from a self-help women’s or- 
ganization known as the Annapurna Mahila Mandal (AMM), which was 
undertaking development initiatives with some of the women, and to whose 
work the results of the study were likely to be relevant. This was not a 
‘fast’ research project. It does, however, illustrate some of the difficulties 
which researchers may have to confront. 


By its very nature, the study was located in the slum localities of Bombay. 
These slums are characterized by crowding, congestion and lack of basic 
facilities. In these conditions, which are often unfit for human habitation, 
people try to eke out a living and survive recurring crises of poverty. 


As with any other outsider who visits such localities, my early reaction to 
what I saw was one of disbelief and hopelessness at the sheer immensity of 
despair. It was one thing reading about poverty — it was another being so 
near it. But I was willing to put myself through the distress I felt because I 
was keen to gain my PhD which required me to carry out an original piece 
of research. Like other researchers, including those more directly linked to 
action but equally likely to be driven by the needs of their job and career 
aspirations, I therefore had a very selfish motive in my consideration of 
poverty. However, what I saw even within the first few days upset me so 
much that I could not handle the idea of research without asking myself 
several questions. What was I doing, how was I doing it, and in particular 
was this ethically and morally acceptable? 


Thinking about such issues was difficult and at times emotional. It required 
a very personal reflection on the whole of the research process and the idea 
of outsiders looking into poverty. But as I will show here, such personal 
reflection is an essential part of the research process, whether that research 
is fast or slow. In fact, I would suggest that it is precisely this type of 
questioning that leads to an enhanced understanding of both the research 
question and the research process. Furthermore, such questioning also al- 
lows us to challenge dominant ideas of methodology and analysis that are 
often inadequate in the context of poor countries. 


To discuss this, I will begin by giving you some more detail of what the 
research was about. This will be followed by a discussion of some of the 
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issues which shaped my choice of methodology, and a brief description of 
the fieldwork. The final section is a reflection on the form of the results 
and some of the ethical questions that were generated throughout the re- 
search process. 


9.1 THE STUDY: THE KHANNAWALLIS (MEALMAKERS) OF 
BOMBAY 

My research was about a group of women known as khannawallis — women 
who make cooked meals for migrant workers. (For the complete version, 
see Abbott, 1993.) Simply told, their story began in colonial times when 
Bombay started to develop rapidly as an important port and an industrial 
base. The city’s labour requirements were at the time, and still are, largely 
met by rural migrants. Early census figures suggest that migration was 
predominantly that of single males from neighbouring states, of men who 
retained their ties with the land and continued to return seasonally to their 
fields (Burnet-Hurst, 1925). 


There are several reasons why this pattern of single male migration emerged 
in Bombay. Firstly, poor rainfall, drought and crop failure, together with 
British demand for agricultural taxes, brought abysmal poverty to rural 
families who were forced to seek cash opportunities in the towns. At the 
same time, they could not afford to let go of the land or neglect it. Migra- 
tion was thus selective and seasonal so that the agricultural activity could 
continue. Secondly, while there were plenty of opportunities in Bombay’s 
thriving ports, docks and the textile mills, the shortage of housing was so 
chronic that even if they wanted to, it would not have been feasible for 
whole families to migrate. The men, married or otherwise, thus migrated 
singly to Bombay. 


When they arrived in Bombay, inadequate housing plus the compelling 
necessity to save forced the migrants to share single room tenements (known 
as kholis), often between thirty to forty of them. Besides a place to sleep, 
the kholis did not provide much. The rooms were bare, and had hardly any 
cooking, washing or other basic facilities. In fact, large numbers meant 
that even the space for a sleeping mat had to be negotiated on the basis of 
shift and work timings. 


At the same time, the working conditions of the textile mills and other 
industries were horrendous. The men had to cope with long hours of in- 
tense labouring in Victorian circumstances. (See Morris, 1965, and 
Upadhyay, 1990, for further detail.) However, they managed to rest them- 
selves for work despite the unsuitable sleeping and housing arrangements, 
but it was a problem for them to find a regular supply of cooked food. 
‘Outside’ food was not really a suitable alternative for home-cooked food 
for various economic reasons such as expense, and social reasons, such as 
caste and religious food taboos. 
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Some of the women who did join their husbands in Bombay saw an oppor- 
tunity to make an income in this, and for an agreed sum began to cook two 
meals a day for the kholi dwellers. The men could have their meals packed 
and delivered to either their kholis or their workplaces in lunch boxes con- 
sisting of three or four containers (tiffins). Or they could come to the wom- 
an’s house to eat their food. Women who provided cooked food (khanna) 
thus became known as khannawallis. 


Throughout Bombay’s history, there have of course been several attempts 
at rectifying the housing situation, particularly during those times when 
persistent congestion and lack of hygiene have resulted in an outbreak of 
disease. Thus in the 1920s when Bombay was engulfed by the bubonic 
plague for a second time, the municipality was forced to make improve- 
ments, particularly in the supply and control of basic water and sewerage 
facilities. Bombay, however, continues to suffer, and for millions of mi- 
grants, and the city’s poor, housing still remains unacceptable. It is of little 
surprise then that in 1994 history repeated itself and brought national shame 
when the plague took hold of Bombay once again. 


The demand for the khannawalli service has therefore continued, and has 
provided a living for many generations as the ‘trade’ is passed on from 
mother to daughter. As in the past, even today, several mills and factories 
of Bombay open their gates at midday to allow cart-loads of tiffins to be 
brought in for the workers’ lunch, and thousands of Bombay’s migrants 
still rely on the khannawallis to supply a home-cooked evening meal. Some 
of the khannawallis go even further and enter into ‘boarding’ arrangements 
for men who might not be able to access kholi accommodation, although 
their own homes are themselves usually no more than slum huts or tene- 
ments. 


Like the single male migrants, the khannawallis are poor, illiterate and live 
in slum localities, all of which ensures them a low social status. Thus as 
one report put it: 


... the very term khannawalli has a derogatory meaning. It insinu- 
ates a socially ignorant woman, plying for a trade, looked down 
upon ... Her work is not recognized. After all, what is so special 
about a woman cooking? Contributing to the family income? What 
a fancy idea! She is only feeding herself and her children. As a 
mother it is her moral duty to do so. 


(AMM, 1990, p. 1) 
Because of this image, my interest in the khannawallis was a puzzle to 


many people in Bombay, including the women themselves. But my inter- 
est was based on at least three aspects: 


1 The khannawallis were and are an essential part of Bombay’s industrial 
development and have historically enabled scores of migrants to live 
and work in conditions that would otherwise have been unacceptable. 
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2. The khannawallis contribute extensively to their family’s income and 
enable survival for both their rural and urban families even if this often 
remains unacknowledged. 


3 In 1975, under the leadership of Prema Purao, a former trade union 
official and Communist Party cadre, the khannawallis who normally 
work in isolation in their homes founded a trade-union-like organiza- 
tion known as the Annapurna Mahila Mandal (Abbott, 1992). The AMM 
functions on similar principles to other ‘grassroots’ organizations that 
have attempted to organize poor women in India, such as the Self-Em- 
ployed Women’s Association (SEWA) and Working Women’s Forum 
(WWF). Its achievements are impressive and like SEWA and WWF it 
is often regarded as an exemplary model of self-organization of women 
in self-employment in the informal sector (Abbott, 1997). 


Self-organization has brought isolated women together, and has allowed 
them to raise questions about the future of their activity (see Examples 9.1 
and 9.2 below for some life stories). The AMM itself was initially inter- 
ested in my research, particularly as it was in the process of promoting 
itself more broadly to women outside its initial membership. Bearing all 
this in mind, I wanted to find out more about the dynamics of the 
khannawalli activity and the detail of how the khannawallis organize and 
make a living from it. 


9.2 SHAPING METHODOLOGY 


9.2.1 A PROCESS OF REJECTION 


In any empirical work the first question that needs to be tackled is of course 
‘Which method?” 


Weiner (1982, pp. 277-320) argued that in India most social science re- 
search, even that undertaken at universities, leant heavily in one direction, 
i.e. that of applied and policy-directed research, rather than academic or 
‘basic’ research. In this, there was a high emphasis on large-scale, statisti- 
cal types of study in order to provide ministers and decision-makers with 
data that they could easily comprehend. Weiner suggested that often, if the 
research raised controversial issues or the ministers in question did not 
agree with it, it was disregarded or ‘scrapped’. 


It is arguable to what extent this remains the case today. However, at the 
time of my research, Indian studies on poverty usually focused on large 
surveys which allowed a quick gathering of data on numerous aspects of 
women’s and child poverty (the two not being seen as separate issues). The 
issue I wanted to explore, i.e. women’s poverty and home-based income, 
required detailed unpacking of complex social relationships, both within 
and outside the household. I felt that a survey method would be too rigid 
for this. 
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Besides this, there were also several questions on how suitable or ethical 
survey methodology was for the context and setting of the investigation. 
Surveys, for instance, usually imply structured questionnaires (Chapter 6), 
which require ‘official’ looking paraphernalia such as note-books, forms, 
and tape-recorders. What I realized was that ‘officialdom’ is regarded with 
hostility and suspicion because slum dwellers are constantly threatened by 
officials from banks, tax offices, loan recovery and so on. At the other 
extreme, they are also under threat from ‘clean up’ programmes that de- 
molish their homes or subject them to forced sterilization (as with Sanjay 
Gandhi’s notorious attempt at population control). My worry was that I 
would also become associated with this type of officialdom and therefore 
would not be trusted. My feelings were confirmed in an incident that oc- 
curred during my first visit when I was talking to a khannawalli and her 
husband. A neighbour came rushing in and screamed, ‘These people are 
very poor, and you have come to make them even more poor — we know 
you are from the Government!’ The questions I was asking and the note- 
book I was carrying made this man suspect that I was a government offi- 
cial, in pursuit of taxes. Luckily the incident was contained when I was 
rescued by the person who had accompanied me. From that day, I did not 
openly write in notebooks or record conversations. I quickly learnt that 
data-gathering cannot be divorced from such political realities and histo- 
ries, and research methods have to adapt to the context and setting, not the 
other way round. 


Secondly, although the type of information-gathering done in structured 
surveys is necessary from time to time, by its very rigidity it can turn peo- 
ple into ‘objects’ if not thoughtfully conducted. In India, the manner in 
which poor people are subjected to survey investigations sometimes leaves 
much to be desired. From what I observed, such surveys often relied on 
inexperienced undergraduates who are bid to ‘gather data’ on poverty, some- 
times in one sitting. They are given questionnaires which can be insensi- 
tive in themselves, and there is often little account taken of settings and 
ethical or gender considerations. Gulati (1991, p. 15), for instance, cites a 
UN-sponsored study on returned migrants. The survey (which was con- 
ducted by a junior team member) asked: 


What were the effects on marital and family relations (on the 
migrant returning home)? Was the adjustment very good/good/bad/ 
very bad? Did migration of one of the spouses lead to: 


1 greater sharing of responsibility by both spouses? 

2 strengthening of family bond? 

3 loss of affection between spouse, respondent, and children? 
4 infidelity or other marital problems caused by separation? 
5 breakdown of family relations? 
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It would appear that, in full view of the small audience that generally gath- 
ers whenever there is an outsider, and the returned husband, the wife was 
supposed to give honest responses to these leading questions, particularly 
the fourth! This kind of insensitive approach leaves one wondering whether 
such surveys are reserved only for the poor. 


Finally, mine was a gender-focused study which had noted the feminist 
critique of dominant methods of sociological research, particularly the sur- 
vey. While I take into account that some women have used a ‘gender aware’ 
survey method (Chandler, 1990, pp. 119-140), I find that the search for 
quantitative data necessitates a ‘rationality’ that is too rigid and one which 
has been called ‘masculinist’. Particularly in a study such as this one fo- 
cused on women in the ‘informal’ sector, it would have been almost im- 
possible (and counterproductive) to feign objectivity and rationality (for 
further discussion on this point, see Section 9.2.3 below). 


Thus I rejected structured survey methods aimed at quantitative data and 
searched for a methodology which would allow people to tell their stories 
as they wished, rather than those I wanted to hear. I decided to adopt a 
more flexible approach, that of an anthropological type of case-study re- 
search based on several semi-structured interviews (Chapter 6) and par- 
ticipation with a smaller number of people (about 50). In this I was 
guided by what I felt sure I wanted to reject rather than my being sure of 
what I wanted to keep! 


9.2.2 A QUESTION OF REPRESENTATION 


Having rejected the survey approach, with its dependence on structured 
questionnaires, in favour of semi-structured interviews with a smaller 
number of people, I was left with another large question: “Who and how 
many?’ I had to choose whom to interview from a vast area amongst peo- 
ple I barely knew. True, I found plenty of careful guidelines on selection 
and sampling methods. But, I also found such methods invariably focused 
on macro-studies and large surveys carried out in developed countries where 
communications and administrative systems are well established. And 
whereas sampling may work well there, I could not imagine it working in 
the places I was going to. 


Trying to find representative samples in congested Indian settings throws 
up numerous difficulties and frustrations. A good example is Elder’s (1973) 
comparative study of educational achievements of 11-year-olds in Uttar 
Pradesh and Tamil Nadu where he attempted random sampling. After ob- 
taining some 400 names and addresses of 11-year-olds in Lucknow (and 
with a growing realization that this represented a sample of higher castes 
and those who could afford to send their children to school), on checking 
the addresses Elder came across what he calls a ‘serious deficiency’. He 
discovered that: 
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... the school register might carry something like ‘Ram Lal, son of 
Roshan Singh, Aminabad Park’. Several thousand people live in 
Aminabad Park. How was one to find Ram Lal, son of Roshan 
Singh? In the end we were forced to abandon nearly three-fourths 
of the Lucknow sample frame because we could not trace the 
addresses. 


(Elder, 1973, p. 125) 


Clearly, formalized registration, statistics and all such sources of informa- 
tion and secondary data cannot be approached in the same way in all coun- 
tries. In fact, Elder ended up by concluding that ‘no sampling frame 
existed for such a universe. Therefore I had to generate one’ (Elder, 1973, 
p. 123). (See also Olsen, 1992, and Bulmer and Warwick, 1993, for further 
discussion on sampling in India.) 


Fortunately, as pointed out in Chapter 6, micro-studies such as mine do not 
require sampling frames. The choice of who, and how many, to interview 
is based on principles of ‘focused interviews’ aimed at progressively gath- 
ering evidence on the key research question: in this case ‘How do 
khannawallis make a living from their activity?’ . It is nevertheless impor- 
tant to think about selection, representation and cross-sectioning. In the 
‘universe’ I set out to study, I too faced many shocks and difficulties that 
would defy anyone arguing for a representational sample, whether purely 
random or stratified. For instance, at a very early stage of my research, to 
my horror I discovered that some of the people I had contacted and in- 
tended to interview after my first visit were simply not there when I re- 
turned some nine months later. Their homes had been demolished and they 
had ‘vanished’. This meant that firstly I wasted a lot of time in trying to 
trace them, and secondly I quickly had to find others who might fit into a 
similar type. 


At other times, in a desperate attempt to ‘complete the picture’, I made 
many efforts to talk with members of particular groups which I felt were 
essential in giving a fuller understanding of the khannawalli story. Some- 
times, however, all my efforts were defeated. For instance, to my contin- 
ued regret, I had to give up on seeking information from scheduled-caste 
men who ate with the khannawallis. (Scheduled castes and tribes are those 
groups declared to be such by the President of India by public notification 
under article 341 (i) and 341 (ii) of the Indian Constitution. They are recog- 
nized as underprivileged groups, and often include ‘untouchables’, e.g. 
New Buddhists, Harijans and so on.) My sample of the ‘client’ group there- 
fore remained highly skewed in favour of the upper-caste clients whom I 
could far more easily access at their kholis or at the textile mills and facto- 
ries where they worked. 


There were a number of reasons for this. Firstly, the lower-caste clients 
were difficult to locate either individually or in groups. It became impossi- 
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ble to trace the men amongst several hundred others who slept and lived on 
footpaths and at railway stations, or to locate them at a workplace because 
they moved daily in order to take whatever work was going. Their only 
permanent daily contact was with their khannawallis, who would not al- 
low me to speak to their clients for fear of ‘frightening them away’. 


Secondly, even if a subsection of this group could be located in their 
workplaces because they had permanent jobs (where entry through trade 
unions and employers was possible), it was impossible to interview them 
before or during work because they were usually employed by the munici- 
pality, cleaning out the streets and lavatories of Bombay. This meant that 
they left for work even before dawn and they shifted quickly and hastily 
between tasks and locations. Thirdly, interviewing after work was equally 
difficult because on finishing work (usually by midday) the men headed 
straight to a liquor supplier with a ‘no-hope’ attitude. Referring to his daily 
routine and drinking habits, one man said to me, ‘I work in shit and I live in 
shit; what else do you want me to do?’. By about | p.m., the effects of the 
liquor and the heat make the men uncommunicative and sometimes hostile 
(even violent) making it impossible to conduct sensible conversation (par- 
ticularly in a situation where as a woman researcher I felt especially un- 
comfortable and threatened). Therefore, I was only able to talk to a handful 
of clients, and like others who have had difficulties of a similar type, I was 
forced to abandon my plans (see Pettigrew, 1981, for a discussion of prob- 
lems of accessibility for female researchers in India). 


Any researcher or practitioner in the fields of poverty and social exclusion 
will no doubt have several such examples which confirm that neat social 
science methodology cannot fit easily into all circumstances. On the other 
hand, there may be no need to force a fit, because these very differences 
and limitations tell you a lot about the reality of the situation. My example 
above, for instance, taught me that even amongst the very poor, social seg- 
regation of the scheduled castes is very strong, and that perhaps my under- 
standing of how this operated would have remained superficial had I not 
been so keen, if not to find a representational sample, then at least to obtain 
some interviews with members of different subgroups. 


9.2.3 SHAPING METHODOLOGY: GENDER QUESTIONS 

Apart from the ‘technicalities’ of the research method discussed above, 
one very important question was how I would deal with gender issues, 
particularly given the importance of gender relations in the specific social 
context I was investigating. There is a growing feminist critique of what is 
regarded as the male domination or ‘androcentrism’ of the research proc- 
ess (Mies, 1979, p. 2). It is argued that there is deep ideological bias in the 
whole of the research process, including the direction of research policies, 
choice of areas for research and research methodology (Morgan, 1981), 
theoretical concepts (Beneria, 1982, p. 120), analysis, and eventually the 
selection process of publication itself (Spender, 1981). 
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There is a call for a re-evaluation of the whole process, particularly the 
way fieldwork is done. Here, feminists criticize the principles of objectiv- 
ity and rationality that have been popular in the social sciences for a long 
time (Warren, 1988, p. 46). It is argued that emotionality, subjectivity and 
recognition of difficulties enables rather than hinders a deeper understand- 
ing of gender relations. In fact, subjectivity may even make women re- 
searchers more sensitive in the field (Nadar, 1986, p. 114) and be of sub- 
stantive and methodological interest in itself. A challenge to dominant 
methodology is thus seen by many feminists as contributing to women’s 
emancipation in the broad sense. For example, Mies (1979, pp. 6-7) argues 
that ‘conscious partiality’ leads to ‘active participation in actions’; that is, 
by her research and her partiality the researcher is actually entering into 
activism that eventually brings about change. 


In India, feminists have also strongly argued for a marriage between aca- 
demic research and activism which they believe is essential if academic 
work is to relate to reality and not remain abstract and theoretical. A good 
example of this type of activist action-based research is the Report of the 
National Commission on Self-employed Women in the Informal Sector 
(Sharamshakti, 1988). Here, a team of activists and academics committed 
to women’s struggles set out to investigate poor women’s situations through- 
out India (a huge and daunting task) by subjective participation in group 
meetings and conversations with groups of women in several different 
working situations. There was no sampling, no issue of verifiability, no 
concealing of emotions and feelings, but simply the telling of the story of 
thousands of women who struggle to survive in poverty. The evidence gath- 
ered was so startling that the Sharamshakti report consequently led to mas- 
sive changes in policy and the way the Indian Government views the strug- 
gles of millions of poor women in India. 


Equally, in my telling of the khannawalli story, I feel there is absolutely no 
way I could have received insight into the rich details of the khannawallis’ 
lives had I been concerned about neutrality and credibility. In fact, it was 
direct identification rather than an objective stance that enabled this study. 
The point at which people began to relax with me or to accept me was 
when I brought out my identity as a trade unionist or my identity as a 
woman (and particularly my involvement with a battered women’s refuge 
at home in the UK). It was only then that they began to ask me questions — 
and several of them! 


Thus in my ‘objective’ role as a researcher I was just an outside observer, 
and quite frankly (as I was reminded a few times) a nuisance. In fact peo- 
ple did not understand what I was doing. However, in my ‘subjective’ role 
as a woman, a mother, a daughter and a wife, I could make friends and 
have proper conversations with khannawallis. Information thus obtained 
was information given in trust, and I learnt all kinds of ‘secrets’ about 
savings, affairs and quarrels to which I would have been denied access 
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otherwise. Feminist critique gave me the confidence to admit subjectivity 
and to reflect on its advantages. 


9.3 CONDUCTING THE FIELDWORK 


This section outlines how information was gathered, mainly from the 
khannawallis and from the clients. In doing so, it discusses difficulties 
encountered in each situation and how these were tackled. 


Like all research, this study also required information from those who were 
not the main actors, but nevertheless helped to complete the picture. 
Amongst others, I interviewed bank managers to understand credit arrange- 
ments for the poor, textile trade unionists to enable a sounder understand- 
ing of the textile history and how changes here have affected the 
khannawallis, and visited relevant women’s organizations located in Bom- 
bay and elsewhere. In addition, I talked to several academics and activists. 


In comparison to khannawallis and their clients, there were no real prob- 
lems encountered in gathering information from these other sources, and 
so this section concentrates on the two main groups of informants. 


9.3.1 INFORMATION-GATHERING FROM THE KHANNAWALLIS 

Who and how many? 

With the khannawallis, as discussed above, there could be no attempt to 
obtain any type of statistically significant sample, and therefore my aim 
was fairly broad: to cover as wide a range of situations as possible. In that, 
I was hoping to talk to at least fifty women initially and choose some for 
more extensive study. 


But, whereas I hoped to cover a wide spectrum, it was necessary for the 
sample to draw out social differentiation between the khannawallis (be- 
cause that was a primary objective in the study). I felt that two sets of 
criteria might enable a comparison between women from various groups: 
(a) differences in their caste/religion/marital status; and (b) differences in 
their association with the AMM, i.e. AMM membership, non-membership, 
and ‘potential’ membership. (Potential members were those waiting to join 
the AMM, usually for six months. For further discussion see Abbott, 1993, 
ch. 8; also Abbott, 1992.) 


I was confident that I could gain access to these women through the AMM 
because initially the AMM assured me of their full co-operation, and I 
knew that the khannawallis would not talk to me without the approval of 
the AMM. Although for reasons unknown to me, certain influential indi- 
viduals in the leadership later withdrew co-operation, I was able to con- 
tinue with help from a number of AMM “area leaders’, who were responsi- 
ble for looking after the khannawallis living in their localities or areas. 
Altogether, I managed to interview 33 AMM members, 12 potential mem- 
bers, and 17 non-members (these figures include two group interviews, 
which occurred when a number of khannawallis lived very close to each 
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other — for example, in a ‘police building’ I visited, where the khannawallis 
had sons or husbands in the police force and were eligible as a family for 
rented accommodation from the police). 


Once the initial visits were made, people were chosen for second and re- 
peated visits depending on which categories they fitted into, and how inter- 
esting their case study was. As far as the categorical choice was concerned, 
there was no hope of selecting on the basis of ratios or proportionality. 
There were, for instance, proportionately far higher numbers of upper-caste 
and married khannawallis simply because (a) until the founding of the AMM 
it was the upper-castes who dominated the khannawalli activity and (b) in 
general, most of the women are married, very often at a young age. Thus, it 
was decided to choose from the predominant categories based on the vari- 
ety each case offered, while all those in secondary categories (such as sin- 
gle women and scheduled castes) were automatically put down for at least 
one repeat visit. Who was visited for the third or fourth time thereafter, 
often depended on how forthcoming the women had been on the previous 
visits. 


A similar format was followed for non-members, although it was far more 
difficult to trace these. However, despite AMM’s warning that I would not 
be able to find any khannawallis who were not associated with them, initial 
introductions ‘snowballed’ and I managed to interview at least seventeen. 
These women were introduced to me by union representatives as well as by 
clients who worked in textile mills. All seventeen lived in the better-off 
localities and appeared to be in a stronger financial position than the AMM 
members interviewed. There was also a higher level of male employment 
in this group. Perhaps because non-AMM khannawallis lack an organiza- 
tional structure, they were more suspicious than AMM members. There- 
fore, the initial interviews were problematic, and on one occasion had to be 
abandoned because of a distinctly hostile atmosphere. However, return vis- 
its were made to those who responded well. Here, there were two particular 
cases which were considered exceptional and interesting. Therefore much 
time was spent with (a) a household that consisted of three generations of 
khannawallis; and (b) a household which catered for at least 70 clients. 


Carrying out the interview 

During the first three or four interviews I carried a very small tape recorder 
with a built-in microphone, thinking that it could be used without causing 
much distraction. It was my intention to record and then transcribe the 
majority of interviews in this way. 


However, taping simply did not work because (certainly during the first 
visits) in every location I attracted a small audience. Friends and neigh- 
bours crowded into small hutments and tenements to see what was going 
on, and took great delight in chorusing answers to anything that was asked. 
It thus became impossible to make sense of the tapes or to distinguish the 
remarks that were being made by the khannawalli herself. 
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My problem, therefore, was twofold: how could I control the interview so 
that I captured the khannawallis’ responses only; and how could I record 
them because (for reasons described in Section 9.2.1 above) I was loathe to 
carry a notebook. Also, of course there was the additional problem of lan- 
guage, when, initially at least, even those fully conversant in Hindi insisted 
on using Marathi, which needed to be translated. (As I was later to dis- 
cover, this was for social and political reasons. Firstly, the khannawallis 
associated Hindi with class and power, which they felt they lacked. Sec- 
ondly, they saw the use of Marathi and a rejection of Hindi as a part of their 
fight to gain state recognition of Marathi as the regional language of Bom- 
bay. The use (or non-use) of language, then, is a complex issue!) 


I was aware of studies that had experienced similar problems in the Indian 
setting (Patel, 1986) and, like these, the only answer seemed to lie in memo- 
rizing both the questions and the answers. But, in actuality, this proved to 
be a highly difficult task, particularly when translating is also taken into 
account. I discovered that I had often forgotten to ask questions or forgot- 
ten the replies given. 


In order to help me memorize the questions, give some structure to the 
conversations, and control the constant interference, I therefore devised a 
‘questionnaire’ (a semi-structured interview schedule, in the terms of Chap- 
ter 6) which was divided into subsections. The headings and the gist of 
each subsection was memorized, and thus, during conversations, I was able 
to concentrate on a particular subsection and generally recall the questions 
that went with it. In this way, the conversations became more structured 
and if I had forgotten anything, a friend and translator (Anita), who accom- 
panied me to most places, usually reminded me. 


When I devised the questionnaire, I left many spaces in order to record the 
answers — spaces which were never filled out in front of the respondent. 
The answers were recalled as quickly as possible and, at the end of each 
interview, Anita and I would head for the nearest cafe where we would sit 
down to fill the blank questionnaire. Between us, we recalled statements 
and checked details. In this way, I was also able to query translation as well 
as work out what was missing and what remarks needed to be explored in 
the following session. 


The question of interference from onlookers was never entirely solved, 
but, as my visits increased, the curiosity dwindled. With each repeat visit, 
therefore, the chance of the khannawalli responding openly and singly be- 
came stronger. What is also important to note is that as the visits increased, 
the nature of our relationship (and therefore methods of information-gath- 
ering) changed. I was invited to participate in activities within the 
khannawalli’s house (such as meals, celebrations), and outside it (such as 
accompanying khannawallis to hospitals, lawyers, their children’s school 
—in fact anywhere where I could be of use to them in filling out forms or 
meeting officialdom). As discussed in Section 9.2.3 above, this meant that 
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although information-gathering became less formalized, it nevertheless 
opened up areas which otherwise would have been closed. 


In addition to interviewing, I accompanied a khannawalli to her 
native village, which gave me a case study in exploring the link be- 
tween rural migrants and urban suppliers as well as networking in greater 
depth. 


9.3.2 INFORMATION-GATHERING FROM THE CLIENTS 


In Section 9.2.2 I indicated one area of difficulty in obtaining information 
from this group, particularly from the scheduled-caste clients. There were 
at least two other major problems: (a) the Khannawallis’ resistance to inter- 
views with their own clients; and (b) entry to a male domain by a female 
researcher. 


With the first problem, whereas I did manage to interview at least 16 cli- 
ents of the khannawallis in the study, the other 34 ate with khannawallis 
who did not take any part in the study. The disadvantage of this was that 
the ‘mismatch’ perhaps misses out on some of the detail about the 
khannawalli-client relationship that may have otherwise been highlighted. 
For instance, from the 16 ‘matched’ clients (some of whom were boarders) 
I learnt the khannawalli-client relationship could be far more complex than 
that of a customer and a supplier. On the other hand, the advantage of 
interviewing the clients away from the khannawallis’ presence was that 
they gave more honest answers (even more relevant when the khannawallis 
came from the same village or were relatives). Thus, I found that clients 
talked more easily about food tastes, quantity, and their general opinions 
about the service. 


Locating other clients at their rooms (kholis) was not a problem, but gain- 
ing access to rooms which are never visited by women was. However, this 
was overcome when I met Vilas, a young man whose mother is one of the 
khannawallis. Vilas took a keen interest in my work and negotiated my 
entry to the kholis as well as acting as my ‘escort’. Additionally, I received 
help from members of the Communist Party who are active in these areas, 


The kholi interviews were slightly problematic in that the timing was al- 
ways wrong. There were always some men heavily asleep and others too 
tired out from work to respond well. However, these interviews were im- 
portant to me because, prior to seeing the full extent of the stressful condi- 
tions that these men lived in, I do not think that I fully appreciated the 
overwhelming demand for the khannawalli service. And, although the cli- 
ents who lived here did not ‘match’ with my khannawailli sample, this group 
gave me invaluable insight into networking and how their relationship with 
the khannawaillis began — not in the urban setting, but in its rural origin. 
The format was to interview groups of men numbering between 10 and 15 
and this was done in four different locations. A third place where inter- 
views were carried out was the clients’ place of work, with a focus on 
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textile mills because the study was located in the textile mill areas of Bom- 
bay. These interviews were fairly easy to conduct in comparison with oth- 
ers because, in both a private and a government-owned mill, management 
provided full access to those workers who received tiffins from the 
khannawallis. 


Therefore, while I am aware that my client sample was not neat, and that it 
was based on an opportunistic approach, I would argue that the mix that 
resulted proved very valuable. To the best of my knowledge, this is the 
only study that has even attempted to analyse the client-khannawalli rela- 
tionship and explore its differentiation, rather than obtain survey data on 
specific variables such as those relating to credit facilitation. 


9.4 RESULTS AND QUESTIONS 


This study was not aimed specifically either at influencing public policy 
(for example, Indian government policy on urban poverty alleviation) or at 
informing public action by other actors (such as the AMM’s membership 
drive or its attempts to mobilize resources). However, it produced results 
which are relevant to such policy issues in several ways. The chapter closes 
by looking briefly at the results and their policy relevance and finally asks 
some ethical questions related to poverty research, particularly in develop- 
ing countries. 


9.4.1 THE RESULTS AND THEIR POLICY RELEVANCE 


The main results from the study were in terms of how to conceptualize the 
khannawalli activity. The semi-structured interviews with khannawallis 
and their clients, and the case studies of Khannawallis, made sense better if 
the activity was thought of as part of an urban livelihood system or sur- 
vival strategy than if it was conceived as a micro-enterprise. This means 
that for the individual khannawalli the activity is just one part of a continu- 
ous struggle to mobilize resources for survival on a day-to-day basis. Equally 
the relationships with clients are part and parcel of general relations of 
obligation and reciprocity in which caste, kinship and urban—tural rela- 
tions mould how people interact. The idea that the khannawalli activity 
forms a separate ‘business’ part of a woman’s life and her relations with 
clients are mediated mainly through ‘the market’ is less helpful for under- 
standing what appears to go on. 


Two of the case studies of khannawallis are included as Examples 9.1 and 
9.2. The first exemplifies some common patterns among khannawallis: 
strong caste and kinship ties with clients; payment for services overlap- 
ping with other aspects of reciprocal relations; changes over time relating 
to changes in other parts of a household’s livelihood system (e.g. male 
employment or unemployment). The other example is an unusual case but 
reinforces some of the points about the mechanisms of the activity and its 
interrelationship with other parts of a survival strategy. 
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EXAMPLE 9.1 A MARRIED HIGHER-CASTE WOMAN (URMILA) 


Urmila’s case history brings out at least two immediate points: the 
importance of the khannawalli activity to family survival in relation 
to male unemployment and the importance of kinship networks which 
enable the activity. Urmila is a 36-year-old Hindu Maratha woman 
who originates from the neighbouring Konkan. She married at 15 
and has three children (a male aged 12 and two females aged 20 and 
15). Like Sumiti (Example 9.2) she is illiterate and cannot even write 
her name. What is immediately apparent on meeting Urmila is that 
she is suffering from poor health. She has very bad eyesight, con- 
stantly suffers dizzy spells and has been receiving electric shock treat- 
ment to cope with depression. The result is that Urmila looks far older 
than she is. 


The family live in a municipal-owned chawl (a single room tene- 
ment) where they qualify for minimal rent which is some Rs15 per 
month. Urmila’s husband gained access to this housing when he was 
employed at a Government textile mill, a job he lost some six years 
ago. Since then he had had one short spell of employment as a ‘com- 
pounder’ with a pharmacist for which he was paid a monthly salary of 
Rs500. There has been no or very little direct cash income into the 
house for a long time. Some two months ago, Urmila’s eldest daugh- 
ter managed to find employment as a typist and she now brings home 
Rs400 per month. 


From what she can remember, the family owes some Rs6000 to rela- 
tives, Rs2000 to one marwari (moneylender), Rs5000 to a second 
marwari and has received four loans of Rs2000 each from the AMM. 
Recently, Urmila has had to borrow another Rs1500 from a pathan 
for medical costs and thinks there may well be other loans that her 
husband deals with. (People in the slums live in dread that they might 
have to resort to borrowing from a pathan. Because a marwari lends 
on the basis of collateral — usually pieces of gold or silver jewellery, 
brass utensils and so forth — which he keeps until both the loan and 
interest are paid off, the pressure is on the borrower to pay up if they 
wish to reacquire their property. A pathan on the other hand, may 
lend without collateral but will charge extremely high rates of inter- 
est and/or collect small instalments on a daily basis. Those who fail 
to meet the payments are often subjected to violence and terrorism by 
the pathan and his henchmen.) 


It would appear that the family has sought income generation from 
the khannawalli trade over a period of time. Urmila’s mother-in-law 
was a khannawalli and ever since her marriage she has participated 
one way or another in the activity. However, the crisis created by 
male unemployment has intensified the dependence on that form of 
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income generation which has so far been regarded by the family as a 
sideline. 


Although it might be argued that Urmila’s connections with the ac- 
tivity may have made it easier for her to intensify and expand the 
activity, there were two major problems: (a) a loss of clientele due to 
external circumstances affecting the textile industry; and (b) in this 
particular case, the khannawalli’s inability to cope with increased 
workloads due to her poor state of health. Thus, when her husband 
became unemployed, Urmila increased her client numbers from two 
to eight, but this has now levelled out to four men. All the four are 
closely related to Urmila (two brothers, one of her nephews, and one 
of her husband’s nephews), and are allowed to board with the family. 
(The use of the term ‘board’ here has to be contextualized within its 
overall setting. Within highly congested situations, boarding simply 
means that the men can expect a space to sleep in.) 


In arranging to provide board and cooked food for these close rela- 
tives, Urmila and her family benefit in at least two ways. Firstly, the 
relatives (as with any other clients) are expected to pay for services 
provided. This includes board, a morning cup of tea, and two cooked 
meals a day, for which they pay Rs250 each per month; any other 
requirements (such as washing of clothes) are expected to be met by 
the clients themselves. Secondly, the clients, in their obligatory role 
as close relations, help the family to stretch their resources. Thus 
Urmila’s brothers buy her and her children clothes, bring her a con- 
stant food supply of whatever is grown in the village, and, twice a 
year, allow Urmila and her children to join them in the village for 
anything from two to four months at a time. There is no proper charge 
made for keeping Urmila and her children for this length of time (as 
in an urban area), but they are expected to work on the family land in 
return. The extent of mutual dependency between the client relatives 
and that of Urmila’s family is reflected in the following reply Urmila 
gave when asked if she ever considered increasing the price charged: 


No, I have never thought about that. They have helped me in my 
difficult times. My eldest brother eats only once, but he still pays 
Rs250 ... they take care of my children’s schooling and lend me 
money when I need it. 


Urmila does not wish to expand the client numbers. She feels that 
with her ill health she will not be able to cook for ‘outsiders’: 
For them, I have to cook no matter what happens. At present if I 
am ill, my relatives manage somehow ... they cook for themselves. 


But, she knows that she will probably have to continue with the 
khannawalli work until (or even if) her husband finds employment, 
and this probably be for a long time. 
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EXAMPLE 9.2 A DESERTED SCHEDULED-CASTE WOMAN (SUMITI) 


Sumiti’s is an upsetting story about a single woman trying to bring up 
two children in the daily threat of male violence. In a struggle to 
survive, Sumiti has tried her hand at several income-generating ac- 
tivities, but she feels that the kKhannawalli activity is good because it 
ensures a daily meal for her children. 


Sumiti is about 30 years old, a Harijan (an ‘untouchable’ caste) born 
in Nainital. She comes from a very poor family of agricultural la- 
bourers. When she was 13, her family accepted a brideprice of Rs1400 
from a 20-year-old man. She was legally married to him and lived 
with him for about a year during which time she received constant 
harassment and beatings from her husband’s family. Sumiti thinks 
that this might have been because she could not work as hard as she 
was expected to in the fields. 


After a year, in the face of financial hardship, her father-in-law ar- 
ranged for Sumiti to be sold into Bombay’s prostitution market. The 
agent who took Sumiti to Bombay raped her and then decided to make 
her his mistress. Thus Sumiti escaped prostitution but was forced to 
live with this man. She was about 14 or 15 at the time when he was 
nearing 45, Sumiti managed to escape one day and eventually found 
a job as a live-in domestic servant. The man, however, succeeded in 
tracing her and ‘persuaded’ her to return to him. She became his mis- 
tress and lived with him until three years ago, having two children by 
him (8 and 9 years of age). During this time, the man bought a room 
in the slums of Siddarth Nagar Worli in Sumiti’s name, and despite 
its locality, the room would now raise a large amount of money (any- 
thing between Rs50 000 and Rs60 000). After years of cohabiting with 
Sumiti, the man decided to return to his wife and their children some 
three years ago and now wants to sell the room. He has not given 
Sumiti any money since he left and, in order to drive her out of the 
room, continuously and daily harasses her and the children through 
extreme forms of physical violence. 


Sumiti feels that her neighbours are encouraging him. They do not 
like her caste, think she is a ‘loose’ woman, and are jealous that she 
has been sending her children to an ‘English-medium’ school. 


Sumiti is in deep debt. She has pawned whatever she could to the 
marwari and has further borrowed money from the AMM. In order to 
survive, she combines a number of income-generating activities. She 
works as a part-time domestic servant, and buys saris at a wholesale 
price and sells them in the neighbourhood for (higher) payments by 
instalments (although she has now stopped doing this because there 
is no money left to make the initial purchase). Five to six months ago 
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she became a khannawalli by letting out the partitioned top half of the 
room to four boarders. Sumiti is very hesitant in relating exactly how 
she has acquired these clients and the only information she imparts is 
that they are from the same village as her. She provides one daily 
meal of chappattis and vegetables for them. In return she receives 
Rs200 a month each from them. Various incidents and data discrep- 
ancies led me to believe that Sumiti had both acquired and retained 
clients by also providing sexual services. 


At the time of the interviews, Sumiti was worried that she was going 
to lose her clients who were thinking of returning to their native vil- 
lage, and was desperate to fill out a passport application in order to 
migrate to Dubai, where she has heard that domestic servants are paid 
Rs800 a month. 


The results from the studies relate to policy fairly directly, in that the com- 
peting ideas or alternative conceptualizations of the khannawalli activity 
have very different implications for policy or public action. For example, 
in terms of provision of credit, a micro-enterprise will have very different 
needs from a poor woman or family struggling to survive. The latter may 
not be helped at all by a policy which ties credit to meeting targets in busi- 
ness plans. However, it must be reiterated that despite the clear policy rel- 
evance, the results of this particular study were not tied in at the time to the 
actions of key policy actors such as AMM or the Indian Government. It is 
possible that such results contribute indirectly to a better understanding of 
urban informal economic activity and hence to a generally better apprecia- 
tion of policy requirements among those responsible for poverty allevia- 
tion or micro-credit provision — but this cannot be proved, and there may 
well be other, stronger influences on how such people conceive of poverty. 


9.4.2 ETHICAL QUESTIONS 

Throughout this research, the most pressing questions for me were those of 
ethics. Two main areas of concern were: (a) that of being a foreign-based 
researcher; and (b) that of commenting on poverty from a position of clear 
privilege. 


To start with the first question, there is severe criticism levelled at foreign 
researchers in India. And, this is understandably so. There is a long history 
of colonial sociological and anthropological research that has always given 
distorted and biased interpretations of Indian society. (See for instance 
Srivastava’s (1991) follow-up of Elwin’s (1939) work on the Bagia tribe.) 
As a result of this history, foreign-based researchers are often seen as rein- 
forcing their higher educational status, and government research and teach- 
ing appointments frequently go to Indians who are trained in British or 
American universities. Sociological research is therefore seen as a political 
arena where ‘western’ ideology and power need to be challenged. This chal- 
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lenge has resulted in a strong critique of ‘western’ theories and ‘western’ 
models with the aim of replacing these by action-oriented research where 
the researcher and the participant work together to enable change more 
suited to their country’s need. (See, for example, Clinard and Elder, 1965, 
for a further discussion on defining characteristics of Indian sociology.) 


Intellectual resentment of foreign methodology then spills over in a per- 
sonal resentment of the researcher who is perceived as having more ad- 
vantages than the Indian researcher. This includes the foreigners’ ability to 
buy space and time because they have more money (even if this is simply 
due to unfair exchange rates) or better access to library computing and 
other facilities. Also, there is an awareness that if work is published in 
Britain or America it will be taken more seriously and gain quicker recog- 
nition than work published in India. Added to this is the question of how 
accurate studies carried out on flying visits might be. 


I cannot say that this type of resentment has not bothered me in my role as 
a researcher, even if I understand the reasoning behind it. The only way I 
could cope with this dilemma was to be aware of it but go ahead in spite of 
it, realizing that I could only narrate the story from a foreign point of view. 
And I determined to make an effort to understand the criticism levelled at 
this type of research. 


To a certain extent, therefore, I have been able to deal with these issues by 
recognizing the limitations of what can be changed and what cannot. But 
the question of being a foreign researcher has, in fact, posed a lesser di- 
lemma for me than that of my ability to comment on poverty. How can I, 
who will never experience the type of poverty that the khannawallis face, 
write about it? And haven’t poor people been research objects for long 
enough? Also, I found it difficult to answer the question “What will you do 
for me?’ — particularly when it was I who was going to benefit personally 
(with my degree, at least) from the research. Even where research is more 
directly policy related, it is not likely that those being researched will ben- 
efit immediately — policy change is more likely to have an indirect effect 
on others like them. 


Nothing in social science had prepared me for dilemmas like this, or taught 
me how to react to poverty. My privileged status was a constant intrusion, 
and in their friendliness and hospitality the khannawallis continued to re- 
mind me of it. So they rushed to borrow a chair from the neighbours when 
I visited, even though I insisted on sitting on the floor; they sent children 
out to buy soft drinks and milk for my tea even when I knew that these 
would never be bought for the children themselves; and they insisted on 
“honouring me’ by cooking a mutton curry even though I am a vegetarian, 
because this is a rare treat for them! 


Through their actions, the Khannawallis made it quite clear that there was 
a vast gap between us and I would never be able to understand poverty 
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except from a privileged position. Therefore, this is what I did, and my 
understanding of poverty came through my own reactions to it, an under- 
standing which shocked me out of any assumptions that I held about being 
poor and which in turn generated numerous other moral dilemmas. 


To give an example, social science has taught me that when conducting 
fieldwork, expect the respondents to ask questions about yourself and be 
prepared to give truthful replies. So, when women asked how many chil- 
dren I had, I replied that I had two. What came to me as a shock was their 
next question ‘How many have died?’ The matter-of-fact tones of this ques- 
tion (which recurred several times), and my own startled response to some- 
thing I would never have dreamt of asking anyone at home, told me more 
about poverty (and my privilege) than any amount of quantitative data on 
the subject of infant mortality. 


In another instance, on a repeat visit to one of the kKhannawallis, I saw her 
seven-year old daughter lying on a mat. I stroked the child’s head and asked 
her what the matter was. Her mother informed me that the child had ty- 
phoid. I had to prevent myself from jumping up and showing my panic at 
the thought that I had actually touched someone with typhoid. Having tried 
to control myself, I asked her if the child received any treatment. The mother 
nodded and showed me a packet of Boots® paracetamol! 


In my supposedly objective role as a researcher, I faced many dilemmas in 
a matter of seconds. I cannot comment on how the mother must have felt 
because I had not come across that type of helplessness before. I could not 
inform the mother that the medicine on which she had spent (what was for 
her) a lot of money would do little for the child. I could not ‘rescue’ the girl 
because I knew that in poverty these things happen over and over again. 
Yet I could read into the mother’s eyes that she wanted me to give her some 
money for the child, and I could not but empathize with her because I have 
a child of about the same age. 


There are no sociological guidelines on how to deal with this, or the guilty 
feeling I developed, because whereas I could walk away at any time, the 
people who had become my friends could not. I have never been comfort- 
able in the role of a ‘surveyor of poverty’, and I could not remain objective 
when I felt angry and distressed at what I saw. Therefore, I could only deal 
with my privileged interpretation of poverty through a practical approach 
that was unwittingly suggested to me by one khannawalli when she said: 
I don’t know why you want to write about me. I am illiterate and 
don’t know much. But, you know how to write. It is your job, so 
write what you have to! 
So this what I did. I wrote about what I saw, and was aware of my limita- 


tions. Working these out was a lengthy process in which personal reflec- 
tion has played an essential and enriching role. 
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The acceptance or rejection of policy often rests on how that policy is jus- 
tified and how the justification is presented. Communication therefore is an 
important aspect of policy-oriented research. Policy issues can be politi- 
cally sensitive, and successful communication of results involves complex 
judgements and a sense of timing. Here we consider a set of related issues 
together with some basic suggestions about how to communicate results. 


Policy investigation often involves finding out about people, organizations 
and people in organizations. Results are prepared for those charged with 
formulating and implementing policy, but will also often need to be com- 
municated to, and approved by, a range of other stakeholders. Communi- 
cating results to those directly affected by the policy may be a particularly 
delicate and important task. 


We put forward two broad perspectives, in Sections 10.1 and 10.2, on com- 
munication of results as part of the research process. For those managing or 
carrying out policy-oriented research, thinking through how to communi- 
cate results itself often involves investigation and planning. Ideally, com- 
munication between researchers and other stakeholders in the project is a 
two-way process: flexibility and some degree of sensitivity to a range of 
different interests and concerns may be necessary. 


In Section 10.1, we argue that communicating results is not something that 
should be thought of as coming at the end of a research project — rather it 
should be integrated into the design of projects from the outset. In this sec- 
tion we point out the benefits which can come from greater and more effec- 
tive communication. 


In Section 10.2 we discuss some of the constraints and political considera- 
tions which in reality often impact on the sequencing and substance of com- 
munication. While ideally communicating results is integrated fully into 
the research process from early on, thus maximizing participation, in some 
instances it may be necessary to think about the impact of research from 
other angles. Some stakeholders may demand that communication is much 
more circumspect. This may, or may not, be legitimate but in any case deci- 
sions about the when, what, who and how of communicating results often 
raise questions about power, ethics, accountability and responsibility. 


10.1 COMMUNICATION AND THE RESEARCH PROCESS 
10.1.1 AN ITERATIVE PROCESS 


Communicating results is an activity which is often thought of as coming at 
the end of a project — you define the project, do the research and present the 
results. This is often not the best way of conceptualizing the process. Good 
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research often requires layers of iteration. Through preliminary investiga- 
tion, it often becomes apparent that the initial design of the project is lack- 
ing, for instance. Alternative research designs will need to be thought 
through. Sometimes the researcher will realize the need to change and will, 
in discussion with other stakeholders, be able to make the appropriate al- 
terations. This process may also lend insight into the explicit and implicit 
motives behind the commissioning of the research. 


Joseph Hanlon provides a good example of this in Chapter 3 (Section 3.5), 
when writing about his work on land issues in Mozambique. Here the 
initial enquiries made it clear that the original brief should be revised. 
In the course of discussions with the commissioning NGO about this, 
the NGO’s agenda became clearer. Hanlon was able to suggest a set of 
revised research questions, based both on his new understanding of events 
and situations in Mozambique and on the NGO's aims and objectives in 
commissioning the research. 


In this case it was the researcher who realized there were problems in the 
way the brief had been defined. In other cases, communicating preliminary 
results to others may bring to light problems which the researcher may 
have missed and which can then be discussed. Thus, one reason for inte- 
grating communication and presentation into the process of research at an 
early stage is that design faults can be dealt with. 


10.1.2 THINKING ABOUT COMMUNICATING CONCLUSIONS 


Thinking about communicating conclusions and presenting tentative con- 
clusions can shape the research in productive ways. The process of reach- 
ing a set of conclusions involves trying to pull out essential elements from 
the work carried out and weaving a coherent ‘story’. Conclusion making 
and testing is a part of conceptualizing and reconceptualizing of investiga- 
tion which Alan Thomas refers to in the Introduction and it often encour- 
ages the ‘two-way relationship between ideas and evidence’. In Chapter 4 
Stephanie Barrientos points out that sometimes it is not until writing up 
that you know if your argument will work and you can provide a conclu- 
sion. In this respect it is the same when you have been thinking about 
people and organizations as when doing a literature study. 


Communication of initial results can also highlight any misunderstandings 
about the methodology being used and how the research is understood by 
those carrying it out. This is particularly important when several people, or 
teams of people, are engaged in the research. A recent project looked at 
risk regulation of biotechnology in several European countries. Although 
early discussions had defined a common methodology, nationally based 
teams carried out research independently of each other. At an interim meet- 
ing, about a third of the way through the project, the research teams dis- 
cussed the progress of the work and how the final report should be written. 
This brought to light several discrepancies in the methodology in each coun- 
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try. For example, the way in which interviews were carried out in different 
places meant that there was a difference in the amount of ‘formal’ statisti- 
cal data that could be used from each survey. While it seemed that the same 
type of information was being collected in each place, detailed discussions 
about presentation of results revealed substantial differences in the nature 
of the data. 


Communicating ideas generated early in a study helps to provoke feed- 
back on the way you are thinking about the topic. If you have an interesting 
or new approach (or even if not), communicating your ideas early (for 
example, in a workshop to a variety of interested parties) may prepare the 
ground. People you are presenting to are likely to respond better to new 
ideas if they are not surprised with a radical approach at the end of a project, 
but have a chance to think through the implications before final rec- 
ommendations or conclusions are made. Additionally, you may find your- 
self being more open to critical feedback before you have become too 
wedded to an approach. 


Philip Woodhouse (Chapter 6) notes the importance of establishing rap- 
port with research subjects. Feeding back tentative conclusions and ideas 
formulated on a work-in-progress basis can be a good way of building 
rapport among different stakeholders, including interviewees. The process 
of presenting or discussing tentative conclusions may well generate new 
information and perspectives. 


10.1.3 PRESENTING RESULTS 

It is helpful to present research findings throughout the research process to 
participants, funders and other stakeholders in order to clarify analysis, try 
out ideas and deepen understanding and knowledge. Such a process can 
often enrich research and provide an opportunity to check approaches and 
results. There are various ways to communicate results including the fol- 
lowing: 

* formal and informal presentations to stakeholders; 

* breakfast/lunch/dinner talks; 

* newspaper/popular journal articles; 


* public workshops with selected invitees from the policy world, experts, 
academics and participants in the research. 


Decisions about how to organize communication will depend on a range of 
factors, including the scale of the project, the relationship between research- 
ers, funders and other stakeholders, the extent to which the research is 
‘public’ and the extent to which participation and feedback from the ‘sub- 
jects’ of research is required. How you choose to structure presentation of 
results will clearly depend on what stage you are at and what you hope to 
get out of the presentation. 
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Constructing a presentation 

The wide range of different fora and circumstances which might be associ- 
ated with communicating results makes it vital that some thought is given 
to how to construct communication. One way of thinking about this is to 
ask some simple questions. Box 10.1 outlines some ‘Who? What? Why? 
When? How?’ questions which may be helpful in thinking through your 
presentation. 


BOX 10.1 START BY ASKING THE FIVE QUESTIONS: 
WHO? WHAT? WHY? WHEN? HOW? 


Who are you communicating with? 

¢ What are their levels of knowledge? 

© What are their roles? 

© Who holds the power? 

* How many are there? 

¢ Are they the right people to communicate with? 
* How are they likely to react? 


What do different stakeholders want? 

© What are their needs, their concerns, their expectations? 

* What matters to them? 

* Consider this before you decide on your own objective and strat- 
egy for handling the presentation. 

Why are you making the presentation? 

What is the purpose of the presentation for you? 

¢ What is your message? 

© What is your objective? 

* What ideas, products, services are you presenting? 

© What kind of feedback are you interested in? 


When will you make the presentation? 

¢ At what stage of the project or process are you making the presen- 
tation? 

* Do you have the information you need to make your points? 

¢ Is the physical space (or document format) appropriate? 

How will you do it? 

* What tools will you use to help put your message across — notes, 
visual aids, documentation, amplification, props, etc.? 


(Adapted from Stanton, 1986 and JSB, 1996) 
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Preparing a presentation 

The presentation of information is often not given enough time relative to 
other parts of the research process. Preparation is essential. It is after all a 
waste of time and energy to spend time investigating and analysing infor- 
mation but then not to take time to make sure that others understand your 
perspectives. 


In addition to the checklist given in Box 10.1, there are a number of other 
simple (but often forgotten) tips which may help you construct presenta- 
tions. 


1 Make a few clear points 
As Hanlon notes in Chapter 3 with regard to written reports, it is impor- 
tant to decide on the main points of your presentation. It is best to stick 
to a few points rather than overload the audience with a lot of detail. 
Questions of detail about the key points can better be dealt with in dis- 
cussion or question time. Three points per presentation is a very com- 
monly used ‘rule’. Many people like to begin a presentation, or written 
report, with reference to the three main points which will be dealt with. 
This is a useful device for structuring the content of your talk or report. 


2 Stick to time 

It is important to let the audience know the length of your presentation 
and to stick to that time. People will appreciate good timing and are 
likely to resent being held up by a presentation which has run over the 
time allocated for it. There are rare exceptions where an important dis- 
cussion is in progress and it is inappropriate to break. However, always 
be aware of time. If the presentation has run over, you could offer to 
make yourself available for further discussion after the group has bro- 
ken up. If you really feel that you should not break up the gathering, 
you should still check with the audience whether they would like to 
continue or not. 


Remember that people cannot concentrate for more than 5 to 10 min- 
utes at a time. If the presentation is likely to be long, you could consider 
ways in which to break the time up. You may want to use a joke or 
gimmick, but do so with great caution. Be sure that any humour or gim- 
mick is culturally appropriate. You may construct the presentations so 
that you highlight key points at various times during the presentation. 
3 Using visual aids 

Visual aids can help focus an audience’s attention and can help you to 
structure written or oral presentations. However, the visual aid can be 
badly misused! An overhead overloaded with information which con- 
tributes nothing to the speaker's efforts to communicate is a common 
sight. 

Presenters also often use overheads as ‘prompts’ for themselves. While 
this can be useful, overheads should also be used to present data and 
arguments to an audience in a more accessible form. The overhead il- 
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FIGURE 10.1 An example of a visual aid to presentation: a framework for 
sustaining business growth (simplified from Segal Quince Wicksteed Ltd., 1995) 


lustrated in Figure 10.1, taken from a report on the needs of medium- 
sized enterprises in Poland, summarizes the consultants’ analysis. Yet 
the same kind of summary would be difficult to achieve in words. When 
consultants used the overhead in a presentation and in a written report, 
particular parts of the diagram were highlighted, and drawn as indi- 
vidual overheads. This focused attention on the particular points being 
made. 


Charts and pie charts can also be a useful means of transmitting infor- 
mation in an accessible way. They may also make you think differently 
about the work which you have been doing. This is because the process 
of putting together a diagrammatic representation of material actually 
encourages you to look at the data, and your arguments, very carefully 
and from a variety of angles. 


It is also helpful to make printed hard copies of slides or overheads to 
give to people if there are particularly important points which you want 
them to remember. 
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When you make a presentation using visual aids, remember to talk to 
your audience rather than to the slide or overhead diagrams. 


Handling questions at a presentation 

If you are making an oral presentation you will usually be required to an- 
swer questions. This often is quite a difficult time and can make the pre- 
senter feel quite vulnerable. The following are a few ‘rules of thumb’: 


* If you don’t know, don’t pretend you do 


A colleague (who incidentally is an expert in communications) recalls 
that at the end of one presentation of research, the funders asked if he 
had surveyed XX (which he assumed was a journal) as part of the re- 
search. Feeling flustered and slightly ashamed that he had never heard 
of the journal, he said he had looked for articles, but had not found any- 
thing of relevance. The client, somewhat cruelly, went on to ask where 
he had located the journal and what years he had looked at. Our col- 
league mumbled for a while before the client laughed and said that the 
source of information he had referred to was a computer database and 
not a journal. 

Most of us will recognize this scenario and feel for the consultant. How- 
ever, you would probably also agree that when asked a question about 
which you have absolutely no idea, it is not good practice to make it up! 
Although it can be difficult at the time, it is better to admit that you don’t 
know. You can follow this with an offer to find out and get back to the 
person who asked the question if this is appropriate. 


¢ Be clear if your answer is a personal opinion 


In many instances there is no clear distinction between opinion and the 
data collected — it is a matter of interpretation. Sometimes, however, you 
will be asked questions that require you to give an opinion. If you are 
uncomfortable about this, say so. You can preface your answer by say- 
ing that this is your opinion. You may want to add that it does not neces- 
sarily represent the views of your organization or others you are work- 
ing with. In some cases it may be better simply to say that you don’t feel 
in a position to answer the question and say why. 


* Don’t condescend or be rude 

Sometimes you will be asked a question which you consider silly or 
obvious. It is always better not to reveal that this is what you are think- 
ing! First the question may appear stupid or inappropriate to you, but it 
may not to others in the audience. Second, if you humiliate someone 
less experienced than you, you are likely to be considered a bully. If you 
humiliate an equal or a senior, you may well suffer retaliation. Third, a 
tude response is likely to be considered unprofessional. Even if the ques- 
tion is considered stupid by everyone, your tact is likely to be appreci- 
ated. 
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© Unpack the multiple question 
If you are asked a question which is complicated and appears to contain 
a number of points, try and get the questioner to break it down. It will be 
easier for you to answer. Another way you can deal with this is to re- 
phrase the question and check with the questioner that this is what they 
wanted to know. If the question is running on for a long time, you can 
intervene, saying ‘Perhaps I could deal with your first point’. 


¢ Keep calm in response to hostile questions 
If you are asked a hostile question, try and respond coolly and calmly. If 
you reply in kind, you are likely to escalate the dispute and it becomes 
harder to control. 


A company document The Presenter’s Guide (JSB, 1996) is useful if you 
require more detail on how to handle questions and presentations. 


10.1.4 INTEGRATION INTO THE RESEARCH PROCESS 


So far this section has discussed various tools and approaches which we 
hope will help you in putting together presentations. The main point of the 
section has been that communication of results is not something which 
only happens at the end of your research or consultancy — rather it needs to 
be built in at various stages of the process. It needs to be viewed as a two- 
way process and you need to be able to respond to input from stakeholders 
if you wish to make your research relevant and useful. The communication 
process itself should also be thought through and constructed so as to make 
the most of the work you have done. 


This view of research and communicating results as a non-linear process 
has implications for researchers, those using the research and funders. 
Among other things, the way in which research is structured and the way in 
which it is evaluated needs to reflect the non-linearity of the process. If 
research is being done in the context of ongoing policy reform, policy ini- 
tiatives may change during the course of the research. Researchers and 
funders will want to build in plenty of space for ongoing discussions about 
the design and progression of the research. A degree of flexibility is desir- 
able, so that time, benchmarks and perhaps even the people working on the 
research project can be changed. If you are doing ‘action research’, making 
changes and examining their impact as part of the research, you will obvi- 
ously need to build in frequent review sessions. In this case, policy and 
implementation will be directly part of the remit. This increases the extent 
to which you need to network with stakeholders and will mean that your 
communication strategy will be an even more integral part of your work. 


Building a pilot phase into the research or consultancy framework can be 
extremely useful. Pilot periods allow funders and clients to assess the rel- 
evance and usefulness of the research question and to amend the design, 
methodology and tools to be utilized in the research. If the research time is 
too tightly scheduled or if the clients and funders are not kept informed 
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about the importance of their input at this early stage, the flexibility of the 
project will be diminished and it may not yield the results required. Re- 
searchers and consultants need to be open and honest about what is work- 
ing and what is not and be prepared to go back to the drawing board if 
necessary. 


However, policy-making is often messy! If communication is not possible 
throughout the life of a project, a whole range of issues may have to be 
addressed. We end this section with an example of a difficult project where 
timing and communication decisions contributed to complications (Exam- 
ple 10.1). 


EXAMPLE 10.1 THE NEED FOR CONTINOUS COMMUNICATION 


In research done for the South African Department of Labour, a lack 
of communication compounded an already difficult and complex piece 
of research. The consultants’ brief was confined to governance and 
financing issues. However, this brief formed part of a broader project 
of the department to write a new policy framework for human re- 
source development. Other researchers, including officials of the De- 
partment of Labour, were investigating a new national, integrated 
system of human resource development and its governance and fi- 
nancing. However, this work only consolidated into a clear frame- 
work towards the end of the consultants’ contract. At that stage the 
consultants were testing tentative policy conclusions with all relevant 
stakeholders. Not surprisingly, there were then major contradictions 
and cross-purposes between the consultants’ recommendations and 
the overall framework. It was clear the consultants needed to go back 
to the drawing board to align, amend or totally reconsider their rec- 
ommendations in the light of this new knowledge. 

Or was it? Realistically, it is to some extent the clients’ (or funders’) 
prerogative to put new information on the table as they see fit, and it 
is the consultants’ (or policy researchers’) job to respond flexibly. 
Yet, sometimes consultants or researchers will be put in a position 
where they have to set limits. Here, an alternative response would be 
to say ‘This was not the brief; these are the recommendations, the 
client should have given us this information earlier; and the consult- 
ants have fulfilled their brief.’ If the client commissioning the work 
is unreasonable, this will sometimes be an appropriate response. In 
the case above, the consultants felt that while decisions about timing 
could have been better, all relevant information had been passed on 
as soon as was possible. They decided that they should try to cope 
with the misunderstanding and spend the extra time to reconsider 
their recommendations in the light of the new framework from the 
client. This proved a valuable lesson for the future both for the con- 
sultants and those commissioning the research. 
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10.2 COMMUNICATION AND POLICY FORMULATION 


Research connected to policy formulation and implementation raises a host 
of issues about accountability, responsibility, power and ethics. A very sim- 
ple example immediately brings to light a number of questions about the 
role of different actors: 


If a researcher or consultant is hired by a NGO to formulate policy 
relating to small-scale farmers, who is the researcher accountable 
to? Is it the NGO? Is it the small farmers? Is it the director of the 
research unit or consultancy firm? Who makes decisions about 
controlling the flow of information? Who should be responsible, 
particularly if things go wrong? 


In practice, the picture is almost always too complicated to give a single 
answer and indeed the aim in most cases should not be to arrive at a single 
answer. However, it is important to understand the complexity in general 
terms and also the implications, in terms of power and ethics, of different 
answers. This section aims to explore aspects of the intricacies of policy- 
oriented research. 


10.2.1 IS RESEARCH AN INTERVENTION? 

In most cases policy-oriented research itself constitutes an intervention. 
By asking people a set of questions, or approaching one group on behalf of 
another, anew dynamic can be set in motion. For example, a research project 
to assess the needs of medium-sized enterprises in Poland entailed exten- 
sive interviewing with the diverse community of support agencies operat- 
ing in the country. The project contributed to an awareness among some 
providers of development assistance and services that more co-ordination 
and communication was necessary. A new set of discussions began as an 
indirect result of the research, independent of the formal progress of the 
commissioned work. Here the research itself acted as intervention and stimu- 
lated a new dynamic among those interviewed. 


This aspect of the research was not anticipated but was welcomed. The 
research began to stimulate activity in the process of interviewing people. 
In a sense, communication of results began there; approaches to people 
signalled that policy was being thought about. In this case some people 
were receptive to the work and began to act accordingly. 


Research ‘subjects’ will clearly respond in very different ways depending 
on their position and context. They may welcome the research, or feel threat- 
ened by it. They may feel that the right questions are being asked or that 
the research is not addressing the right subjects. Whatever the response, it 
is often the case that an approach by a researcher doing policy-oriented 
work (especially if the research is commissioned by a policy-implement- 
ing body) will generate concern and interest on the part of those being 
asked to participate in the research. The way researchers choose to handle 
this can have far-reaching implications for the final outcomes, as shown in 
Example 10.2. 
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HOW RESEARCH CAN PROVOKE 
UNEXPECTED INPUTS 


As mentioned in Example 10.1, the South African Department of 
Labour recently commissioned a new human resource development 
strategy. For the section on the implications of this strategy for train- 
ing providers, the consultants decided to bring together different 
groups of providers half-way through the brief to present their tenta- 
tive proposals. Each group of providers had not only an opportunity 
to hear the proposals long before they came out in an official govern- 
ment paper, but were able to comment, criticize and amend the pro- 
posals from their point of view - which was often in competition 
with another group of providers. The training providers were unani- 
mous in their appreciation of this unique opportunity (never afforded 
them by the previous South African government) and used it to carry 
out a substantial assessment of what would be in the best interest of 
human resource development rather than simply how to secure more 
protection for themselves by privileged access to government funds! 
The feedback it gave to the consultants led them to revise some of 
their initial ideas and proposals. For instance, instead of an initial 
proposal to protect a small group of state-aided training providers 
through an extension of long-term capital funding, they chose to pro- 
vide for differential contracts to any training providers, thus allowing 
any particular training provider to access longer term funding. 


One of the most important implications of this analysis of policy-oriented 
research as an intervention is that researchers require an awareness of the 
nature of power in their work. Much of that power is to do with how re- 
search, and particularly conclusions, are presented to funders. 


In the example of the human resource development strategy for South Af- 
rica, each principal issue — such as competitive tendering — was outlined in 
a number of different ways, allowing the department itself to assess the 
advantages and disadvantages of each and to choose the most appropriate 
recommendation. If the consultants had chosen to make a series of recom- 
mendations with no alternatives, they would have run the risk of being 
highly prescriptive and not giving the client the opportunity to assess a 
range of options. While this could have left the consultants feeling more 
powerful, it would also have meant that the brief which required policy 
options (plural) guided by rigorous research was not being fulfilled. The 
chosen approach still allowed the consultants to choose a particular angle 
to the series of options. The consultants clearly favoured a demand-led, 
competitive-tendering model for the use of public funds for training rather 
than the current focus which is supply-driven training through state-aided 
training institutions. This is a fundamental choice and indicates the power 
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of the policy researcher (the consultants). But in the end the consultants 
had to admit that decision-making power lay elsewhere; it would have been 
an abuse of that power if each recommendation was not evaluated for its 
strengths and weaknesses against alternatives provided within the overall 
framework chosen. It would not have been appropriate to present the pre- 
ferred set of options as the only ones. 


In Chapter 3, Hanlon has a rather different suggestion about how to present 
options in a similar situation. He says ‘If the investigator has a clear rec- 
ommendation but there is a strong opposing view, then the summary should 
only make the case for the investigator’s view, while the main report should 
also clearly explain the opposing line’. While not disagreeing with Hanlon’s 
approach, in this chapter we are emphasizing the need for transparency. 


10.2.2. WHAT ARE THE DANGERS OF RESEARCH BECOMING PUBLIC 
RELATIONS? 

Although there are always exceptions, most researchers and consultants 
pursue their work with a sense of the importance of accuracy and honesty. 
The researcher is often likely to be the person most intimately connected to 
the details of the project and will want long fieldwork hours reflected in 
the outcome of the project. It can therefore be discouraging, to say the 
least, if other stakeholders are uninterested in the process of research. 


In some cases research is commissioned and formulated with a very clear 
idea of the outcomes. Research is carried out with political objectives be- 
ing of primary importance. Although this can be seen as manipulative, it is 
not always a bad thing and does not always have negative results. Re- 
search which has more or less predictable outcomes but is useful at a par- 
ticular time, or in a particular place, can be useful in arguing a point or 
gaining support for certain policies. As long as the research is genuine and 
accurate, this approach is not a problem. Research into the state of the 
manufacturing sector conducted through the South African Industry Strat- 
egy Project aimed to show the extent of the problem in order to identify a 
set of policies in the areas of trade, competition, human resource develop- 
ment and technology. The fact that South African manufacturing was per- 
forming poorly was well known, but the nature of that performance, the 
depth of the problems and the implications for the general economy were 
not agreed upon. In carrying out tightly defined studies for each subsector 
of manufacturing, the Industry Strategy Project (ISP) was able to gain sup- 
port for supply-side measures to be undertaken by government. The cur- 
rent South African Minister for Trade and Industry, Alec Irwin, was asked 
‘What impact did the ISP have on government?’ He responded ‘It’s the 
decisive position that’s been taken. That’s what makes South Africa a little 
unusual in the world economy at the moment. It was the decisive basis for 
forging a trade and industry policy that is compatible with the broader 
RDP [Reconstruction and Development Programme]’ (Irwin, 1996). Re- 
search in this case was clearly carried out with a potential agenda in mind. 
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Nevertheless it was also undertaken with rigour and the findings and analysis 
were based on detailed investigation. 


The dangers are that research is not undertaken in a genuine spirit and 
‘results’ are concocted with reference mainly to desired ends and outcomes. 
There is a difference between commissioning a piece of research for purely 
political purposes without regard for the validity of results and commis- 
sioning serious research at a particularly advantageous time. In practice, 
the issues are often not this clear cut. Sometimes, although the end ‘public’ 
outcome of a piece of work may appear to be a gloss on reality and a piece 
of PR, commissioners of the research may have taken on board the more 
serious research findings. For example, if a report is very critical of a project, 
the promoters of the project may be reluctant to have a negative report 
made public, but may in private be willing to rethink their approval. Re- 
searchers and consultants draw their own ‘moral codes’ around their prac- 
tices. Researchers and consultants are interested in accuracy not only to 
satisfy their own moral codes but also because in the long run it is not good 
for individual reputations if results are consistently seen to be concocted 
for political purposes. People who do not retain a sense of integrity in their 
work are not seen to be credible. Few funders want to hire people whose 
work is not considered as legitimate. 


10.2.3 1S IT EVER TOO EARLY TO RELEASE RESULTS? 

Sometimes the timing of when to release results also presents difficult 
choices and raises questions about accountability. An example from South 
Africa demonstrates some of the difficulties which sometimes confront 
policy-makers when they think about how to strike a balance between open- 
ness on the one hand, and effectively completing the brief on the other 
hand. The Labour Market Commission (LMC) in South Africa — a presi- 
dential commission and therefore publicly accountable — decided at a cer- 
tain stage of its deliberations to stop communicating with the public via the 
press. Its logic was twofold: first, that the expectations of the public were 
growing and that releasing information piecemeal exacerbated this; sec- 
ond, the differing positions of the commissioners would be made public by 
such comments ‘along the way’ rather than the commission utilizing its 
time to develop consensus positions. Being open with the press would cre- 
ate an opportunity for different ideological views to be fought out in public 
allowing different stakeholders (particularly business and labour) to ‘score 
points off one another’. 


This example clearly raises important issues of public accountability. Al- 
though the LMC was a publicly accountable body, the decision was made 
that it could best serve the public by handling its communication in a more 
circumspect manner than it had done previously. There are, of course, po- 
tential pitfalls to this strategy. The LMC could well have been accused of 
being secretive, of manipulating results and conclusions away from the 
public eye. However, it is certainly a political reality that stakeholder views 
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(as reflected by the various commissioners) on labour market policy are 
diverse and at times contradictory. The final outcome of the LMC’s work 
was a consensus report which may not have been possible otherwise. In 
some cases the issue can legitimately be identified as one of controlling 
communication of results in order not to obscure the process of research 
and policy formulation. In other cases, however, lack of communication 
on the part of the commissioning body can mean a more worrying attempt 
to ‘package’ research. 


There are, of course, no set rules for dealing with situations such as these. 
Decisions will depend on political judgement and the stance taken by the 
different individuals and stakeholders involved. This is clearly a very deli- 
cate and sensitive issue. A decision not to communicate should not be taken 
out of any individual motive but rather on the basis of what will be best for 
the final outcome. In this way the project leader may need to play a leader- 
ship role in winning the support for this strategy from the team. Individual 
researchers on the team will need to be clear about why this strategy is 
being adopted and have the opportunity to voice their dissent if necessary. 
There are of course always dangers in this more democratic approach. A 
decision not to communicate could be undermined if one of the research- 
ers on the team decided not to abide by the decision. This would exacer- 
bate whatever conflict situation existed and allow the public/media to fo- 
cus on this dissension rather than on the content of the reports. This media 
focus could then bring into question the legitimacy of the research. Never- 
theless, the underlying principle of discussion among the main groups and 
individuals involved is a very important one and may well, in the longer 
run, be the least ‘dangerous’ of all. In general, if people feel they have had 
a stake in decision-making, they are less inclined to undermine decisions. 


10.2.4 IS NOT COMMUNICATING RESULTS EVER JUSTIFIED? 

In some situations it may suit researchers or funders of research not to 
communicate results. This can lead to difficult ethical considerations. One 
such scenario is when interviewing happens ‘off the record’. As a number 
of other chapters in this book have said, often the most valuable informa- 
tion is given by informal sources who do not wish to be formally quoted. 
There are often good reasons for this. For example, when one of the au- 
thors of this chapter was researching the dynamics of workplace change in 
a small appliance factory in Kwazulu, Natal, many hours of interviews 
were conducted with the shop steward committee. They were keen to give 
the researchers a side of the story that they believed management would 
omit but were also adamant that management should not be told of their 
views for fear of reprisal. Management were convinced that their produc- 
tivity improvements were due to worker motivation and willingness to co- 
operate, whereas the shop steward committee believed this was due to 
workers’ sense of insecurity and desire not to lose their jobs — they did not 
trust management. 
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This story is of interest in that here the researchers were trying to under- 
stand the conditions under which South Africa’s manufacturing industry 
would become more productive so as to withstand competition from the 
international environment. The request from the union produced a dilemma. 
If the researchers did not communicate these results back to management, 
the productivity increases may have been unsustainable in the medium- 
term and an opportunity to put in place the conditions necessary for these 
initiatives to be sustained may have been missed at enormous cost. On the 
other hand, if these results were communicated back to the management, 
the researchers would be betraying the trust of respondents and opening 
themselves up to being blacklisted by the union community from this kind 
of research. 


Eventually, a decision was made to communicate the general lack of secu- 
rity of the workforce back to management without being explicit about the 
fears and concerns raised, in so far as they pertained to the union. Like 
Hanlon in Chapter 3, we would stress that confidentiality is essential. In 
the end, the situation remedied itself, by the union re-establishing itself in 
the factory once the new facility was opened. 


This example highlights a number of issues, including the importance of 
acknowledging power relations between informants and researcher or in- 
vestigator or between different sets of informants and the expectations of 
one set of respondents about the influence of the researcher/investigator on 
another set of respondents. It also highlights the importance of recognizing 
that respondents may be viewing an investigation from a very different 
angle from the investigator. 


There may well be other circumstances where you or other stakeholders 
may view the best public communication strategy as silence — temporarily, 
at least. It has long been a principle of those advocating responsible jour- 
nalism that news reports which increase the vulnerability of already vul- 
nerable groups should not be released. The same principles need to be 
adopted in policy research which is taking place in violent or emergency 
contexts. 


Woodhouse (Chapter 6) and Johnson and Mayoux (Chapter 7) point to 
several of the pitfalls which can be associated with participatory methods 
of research. Some of these points relate more generally to issues of com- 
munication. For example, even good research may be used by more pow- 
erful groups to sanction their own demands. It may also expose but not 
deal with conflict. These circumstances do not necessitate a strategy of 
silence or legitimate not communicating research, but they do highlight 
the importance of carefully constructing results and being aware of the 
impact they may have. 
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10.3 CONCLUSION 


One of the points from this chapter is that the term ‘results’ can be a mis- 
leading way of describing the output from investigation. ‘Results’ tends to 
imply an end point, but it is often more productive to think about commu- 
nicating findings on an ongoing basis. There are many ways in which re- 
search and investigation can be understood differently by the different ac- 
tors involved, or can miss the point without those being involved realizing 
that their work is not focusing on the appropriate target. This view of com- 
municating results as an iterative process has implications for the way in 
which research is structured. Time must be built into the research process 
for reflection and communication about findings. This can be built in as 
‘pilot projects’ or interim reviews, or an ongoing process of meeting with 
different stakeholders in the investigation. 


We have put forward the view that policy-oriented research is itself an 
intervention. In development management, interventions often involve many 
actors who can have very different levels of access to resources. Research- 
ers and consultants enter into relationships with multiple stakeholders, and 
issues to do with power, ethics, accountability and responsibility need to 
be examined. There are many questions which involve consideration of the 
researcher or consultant’s ‘place’ in the research process and who is ac- 
countable and responsible to whom. 


We have not presented definitive answers to these questions. Contexts are 
too diverse and complex for formulaic answers. Nevertheless we have raised 
and discussed some of the questions which researchers and consultants 
may have to confront and suggested some ways to think about responses. 
Communication is an essential part of the research process and learning 
how to communicate more effectively and thinking about some of the rami- 
fications around issues of communication will, we think, improve your 
research and consultancy efforts overall. 
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11 THINKING WITH QUANTITATIVE DATA 


CHANDAN MUKHERJEE AND MARC WUYTS 


Quantitative data play a key role in policy analysis. Perhaps you may think 
that this is so because numbers confer precision. Precision of measurement 
certainly matters and, in some cases, this is the main objective of a quanti- 
tative exercise. For example, a minister of finance may wish to know what 
the likely increase in tax income will be if import duties are increased. 
Similarly, an NGO may want to estimate the expected future revenues of 
one of its income-earning projects. In such cases, precision of estimation 
matters but the result depends greatly on the assumptions made in arriving 
at those estimates. The apparent precision, therefore, can frequently be rather 
elusive (Wrigley, 1988, p. 122). Much depends on how the problem is con- 
ceptualized and on what assumptions are made to arrive at estimates. 


This chapter deals with the relationship between ideas and evidence and, 
in particular, with the role quantitative data play in substantiating ideas. 
Evidence matters because, as Perkins put it, 


... you make a big deal of evidence because you know how easy it 
is for people to feel comfortable with an idea without evidence ... 
and how easy it is for people to think only of evidence that supports 
their viewpoint and not evidence that works against it. 


(Perkins, 1995, p. 332) 


These twin dangers of upholding ideas without evidence or, more com- 
monly, defending ideas only with evidence which supports them, are un- 
doubtedly prevalent in policy analysis where ideas in turn support the na- 
ture and direction of public action. In this chapter, we shall argue for a 
contrastive approach to data analysis which compares rival explanations 
(ideas) in the light of the evidence each brings to bear on the problem at 
hand. In the terms used in Chapter 1, we are looking at alternative possible 
answers to ‘Why?’ questions about the field of policy. The purpose of such 
contrastive inference (Lipton, 1991) is to arrive at that explanation which 
is most plausible in the light of the overall evidence. Evidence consists of 
both quantitative and qualitative data. Other chapters in this book, particu- 
larly Chapter 14, use a similar approach with respect to combining all kinds 
of evidence. In this chapter, we shall deal particularly with the role quanti- 
tative data can play in choosing between rival ideas, although some of the 
points made below apply equally well to qualitative data. 
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Unfortunately, many policy analysts and policy makers suffer from data 
phobia and surrender without a fight when confronted by an army of num- 
bers or formulae supporting a particular policy option. Yet technical quan- 
titative skills which are not backed up by a sharp conceptualization of the 
problem at hand may look impressive, but often rest upon shaky founda- 
tions. It is important, therefore, to acquire numerical literacy to make sense 
of numbers and be able to unpack quantitative arguments and subject them 
to scrutiny. This is not always easy: only careful study and a lot of experi- 
ence will do the trick. But it is also true that quite a lot can be done with 
relatively simple techniques. This chapter gives you some examples and 
develops some principles which can help you in the task of questioning 
quantitative data. 


Section 11.1 discusses the relation between ideas and evidence. While many 
researchers hold the view that proper scientific analysis consists of testing 
an idea in isolation against its evidence, we shall argue that empirical analy- 
sis is essentially contrastive in nature and consists of assessing rival ideas 
in the light of the evidence each brings to bear on the problem. This way, 
the relative strength of different ideas, their loose ends (that is, evidence 
which works against them), and their reliance on ad hoc justifications to 
guard against such loose ends, become more apparent. We shall further 
argue that data assume a dual role in analysis: testing ideas against data 
versus getting ideas from data. Both elements are important and tend to 
reinforce one another in the process of research. 


Section 11.2 asks the question as to what extent data are ‘objective facts’. 
With E. H. Carr, the historian, we shall argue that ‘a fact is like a sack — it 
won't stand up till you’ve put something in it’ (Carr, 1990, p. 11). Data 
always yield a selective view of an aspect of reality and, therefore, what is 
considered to be a fact depends in part on the criterion which underlies the 
selection of data. Subsequently, we briefly discuss the difference between 
primary and secondary data and argue that this often shapes the attitude of 
researchers towards the ‘hardness’ of their conclusions. Finally, we argue 
that certain types of data (in particular, official secondary data), once they 
appear on the scene, often acquire a life of their own and end up exerting 
great influence on the nature of policy debates. 


Section 11.3 uses a concrete example to show how conceptualization and 
technical skill can work hand in hand in questioning secondary data so as 
to bring out patterns within the data which may help you assess rival 
arguments. The numerical skills involved in this exercise are deliberately 
kept minimal: adding, subtracting and calculating percentages is all that 
is needed. Finally, Section 11.4 draws the lessons from this example 
and emphasizes the main argument of this chapter by examining the re- 
spective roles of conceptualization and of numerical skills in quantitative 
analysis. 
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11.1 IDEAS AND EVIDENCE 

11.1.1 THE CASE FOR A CONTRASTIVE APPROACH IN ANALYSIS 

Policy analysis seeks to understand reality, with the explicit purpose of 
informing public action aimed at changing this reality. Analysis has to be 
focused to come to grips with the essence of the problem at hand. Ideas 
help you to do this in so far as they offer tentative explanations for the 
problem. They are powerful since the understanding they yield can inform 
public action, which can change the conditions in which people live or 
work. It matters a great deal how you analyse a problem since it will shape 
the way you perceive the role of public action. 


As a policy analyst, therefore, you should always be aware that your analy- 
sis could be seriously defective or plainly wrong. The problem, however, 
is that the stronger you hold on to a particular idea, the less open you are 
inclined to be to rival ideas. Indeed, the strength of an idea — that it allows 
you to see things in a certain light — also entails its weakness — it often 
prevents you from seeing them in any other way. So, unless you distance 
yourself somewhat from an idea you firmly hold, you will be unable to 
assess the strengths of rival ideas. Perkins gives the following example to 
illustrate this point: 


You enter a rather crowded lounge. You spy a comfortable chair 
and go over but see that someone has spilled a drink on the cush- 
ion. You look around but find nothing to wipe up the mess with, 
and anyway the cushion would still be damp. So you resign your- 
self to sitting on a nearby coffee table. Two minutes later someone 
comes along, takes a look at the same cushion, turns it over, and 
sits down. Why didn’t you think of that? 


(Perkins, 1995, p. 161) 


Does this mean that you should be suspicious of holding firmly to any idea 
whatsoever? Quite the contrary, because doing so would lead to inaction. 
An idea is needed to guide you in a certain direction. What this simple 
example teaches you, however, is that, as with many other things in life, it 
is often the case that in order to advance, you first need to retreat. To check 
the firmness of the idea you hold and to be able to correct for errors, it is 
best to confront it with rival ideas and see how well it stands up to the 
insights they bring to bear on the problem. 


This is where evidence enters the scene: evidence allows you to judge how 
to choose between rival notions. A contrastive approach to policy analysis 
compares ideas in the light of the evidence each brings to bear on the prob- 
lem. You will have seen such a contrastive approach throughout this book, 
and it applies equally to data analysis as, for example, to a literature study. 
In Chapter 4, Stephanie Barrientos argued that you should not only look at 
arguments in isolation, but compare them through analysis. In a similar 
fashion, it is not sufficient merely to look at an idea and its evidence in 
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isolation; you should also compare ideas in the light of the joint evidence 
they provide. In this way, the loose ends and the awkward corners of an 
idea come to the fore as well as the things it omits to consider. 


A contrastive approach to data analysis also guards you against the com- 
mon danger of merely collecting evidence in favour of your pet idea under 
the pretence that you are subjecting your hypothesis to severe testing. In 
fact, if your idea is at all plausible, it is generally not very difficult to find 
some evidence to substantiate it. There are two points to remember. First, it 
is important never to dismiss evidence which goes counter to a notion you 
hold firmly. This kind of counter-evidence may point you towards a stronger 
notion. Second, the fact that an idea is supported by some evidence does 
not mean that the same evidence may not support other ideas equally well. 
It is indeed not difficult to see that a piece of evidence which supports 
every idea under consideration is no evidence at all. It is worthless because 
it does not allow you to discriminate between ideas. So, evidence only 
really matters when it strengthens a particular notion, while at the same 
time going against a rival notion (Miller, 1987, p. 176). 


Let us illustrate these points with two examples. The first example deals 
with a piece of counter-evidence. Suppose you firmly hold the view that 
famines are due to severe shortfalls in the availability (or the supply) of 
food and that you are now confronted with the evidence that food was 
being exported from a famine area when people were starving. This is a 
puzzling fact in the light of the hypothesis that famines result from an 
absolute shortfall in the supply of food. In other words, this piece of evi- 
dence goes counter to the notion held by the researcher. How strong is the 
evidence? Historically, the export of food from a famine area is not a rarity, 
but happens to be ‘... the widely observed phenomenon of what has been 
called a “food counter-movement” by which food moves out of famine- 
stricken areas to more prosperous lands’ (Sen, 1990, p. 380). In fact, *...this 
has occurred in many famines, such as the Irish famines of the 1840s, the 
Ethiopian famine in Wollo in 1973, and the Bangladesh famine of 1974’ 
(ibid.). So how should you deal with this evidence? 


One strategy is to ignore this evidence altogether and discard it as a nui- 
sance. But this does not resolve the issue that this evidence goes counter to 
the hypothesis that the supply of food is all that matters in explaining fam- 
ines. A more fruitful approach is to examine this evidence in the light of 
rival ideas. An alternative notion, for example, is that it is not the availabil- 
ity of food, but access to food, which matters in explaining famines. Using 
the latter idea, Sen explains how food comes to be exported from a famine 
area as follows ‘... there is nothing terribly puzzling about this, since non- 
famine regions (England in the case of the Irish famines, Addis Ababa and 
Asmara in the case of the Wollo famine, India in the case of the Bangla- 
desh famine) often have greater purchasing power and greater pull in 
comparison with the famine-stricken regions, and as a result food can eas- 


240 


11 THINKING WITH QUANTITATIVE DATA 


ily be attracted away through the market mechanism from the famine re- 
gions to non-famine areas’ (Sen, 1990, p. 380). In this case, therefore, the 
evidence points towards this rival explanation: the importance of access to 
food. 


This does not mean that you should now conclude that the variations in the 
supply of food are irrelevant to the understanding of famines. On the con- 
trary. There may well be some truth hidden within the notion that the sup- 
ply of food matters. The argument that supply is all that matters, however, 
is clearly insufficient. But it may well be true that changes in the availabil- 
ity of food (due to, for example, a drought or a flood) matter a great deal in 
so far as they dramatically affect the access to food. In other words, grave 
failures of access to food may well be triggered off by the effects of sharp 
changes in its supply on the way markets operate. But this would lead you 
to pay greater attention to market mechanisms and the way they affect 
access to food, and not just to the changes in the aggregate supply of food. 


There is a further point. Few researchers abandon a theory because some 
evidence goes against it. In fact, few theories are foolproof. There are al- 
ways some awkward loose ends sticking out of any explanation on offer. 
What makes researchers decide to abandon an idea is that a better idea is 
available, and not that some piece of evidence contradicts it. This is the 
reason why you should look at evidence in the light of competing ideas. 


This example also illustrates a final point made earlier in this chapter: 
namely, that quantitative data are not just concerned with precision. The 
issue here was not to arrive at precise estimates of how much food was 
exported from famine areas. Obviously, the more precise such estimates, 
the better they serve the purpose. But it is sufficient to know that signifi- 
cant quantities of food were exported. What matters is that this piece of 
quantitative information allows you to compare rival notions which seek 
to understand the mechanisms of famines. Each of these rival notions points 
towards different ways of dealing with famines through public action. It is 
a fallacy to believe, therefore, that quantitative data only give you quanti- 
tative insights to do with the precision of estimation. More often than not, 
quantitative analysis helps you achieve greater clarity in terms of under- 
standing the problem at hand which, in turn, guides public action. 


Let us now move to our second example which deals with a case where the 
evidence supports rival notions equally well. Table 11.1 shows the relation 
between the average size of land holdings (in acres) and household size for 
a sample of 600 Tanzanian peasant households in 20 villages (Collier, 
Radwan and Wangwe, 1986, p. 50). The authors argue that in Tanzania 
*...other than their own labour, Tanzanian peasant households have only a 
limited range of assets’, of which ‘undoubtedly, the major assets are land 
and livestock’, but ‘in most of Tanzania land is abundant and so we would 
expect its distribution to be determined predominantly by the availability 
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of household labour’ (ibid.). The data in Table 11.1 are subsequently listed 
as evidence for this hypothesis that land size is determined mainly by house- 
hold size. Take a good look at the table to check whether you agree that the 
evidence points towards the explanation put forward by the authors. 


TABLE 11.1 AVERAGE FARM SIZE VERSUS HOUSEHOLD SIZE 


Average land size (acres) Mean household size —_% of households 


Source: Collier et al. (1986, p. 50) 


Undoubtedly, you will agree that the authors’ hypothesis is indeed plausi- 
ble in the light of this evidence; that is, the evidence does not go counter to 
the hypothesis. The point, however, is whether this is the only hypothesis 
that the data support. The authors did not explore alternative plausible ex- 
planations but merely gave the evidence in support of their hypothesis. 
Can you think of any other plausible hypothesis which fits the data just as 
well? Think about this for a moment before reading on. 


A plausible alternative explanation could be based on the following argu- 
ment. The authors assumed that household size is given and, hence, land 
size adapts to the number of hands available and mouths to feed. But is it 
correct to see household size as given? Perhaps the authors reasoned that 
household size depends on the number of children a couple chooses (or 
happens) to have, and, therefore, in this sense is given. But, even so, chil- 
dren may leave a household for a variety of reasons and, similarly, others 
(mainly, close relatives) may join the household. That is, some households 
fragment easily; others remain cohesive. So what then determines house- 
hold size itself? Is it just a matter of chance or, perhaps, does it depend on 
the assets a household holds? For example, a poor household may be more 
subject to fragmentation: children leave home in search of better fortunes 
or are sent to live with better-off relatives. Similarly, a rich household may 
not only be more cohesive (i.e. less subject to fragmentation) but may also 
be able to draw upon the labour of poorer relatives. It is plausible, there- 
fore, that assets (including, but not only, land) matter, even if, on average, 
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Tanzanian peasants have few assets at their disposal. Differences in own- 
ership of assets among households may well be a determining factor which 
explains differences in household size. 


There may be further reasons why household size may not be unrelated to 
the socio-economic condition of the household in question. Take, for in- 
stance, the question of infant mortality. The survival of children into adult- 
hood may differ between poorer and richer households. Furthermore, little 
is known about any differences in fertility behaviour between peasant house- 
holds of different socio-economic status. 


The point is that household size, therefore, is not necessarily given, but 
may depend on the assets a household holds. The implication of this argu- 
ment is that household size may not be a determinant of land size, but that 
household size, and (possibly) land size as well, may depend on the assets 
(including land where it is scarce) a household has at its disposal. The data 
in Table 11.1 support this hypothesis equally well and, hence, do not allow 
you to discriminate between these rival explanations. To do so, you would 
need additional data, possibly including qualitative evidence about indi- 
vidual cases (see Chapter 14). 


The important lesson here is that when a piece of evidence supports an 
idea, it does not mean that this is the only idea it supports. A piece of 
evidence which renders your idea plausible is not much use if it renders 
other ideas equally plausible. The only mileage you get is that it does not 
contradict your idea, but the same is true of rival notions which are equally 
supported by this evidence. Evidence really matters when it allows you to 
discard certain ideas in favour of others. Once more, this example points 
towards the importance of using the principle of contrastive inference to 
guide your research. 


11.1.2 TESTING IDEAS AGAINST DATA VERSUS GETTING IDEAS FROM DATA 
This leads me to the second theme of this section. Data assume a dual role 
in this contrastive perspective: at times, you test ideas against data, and, at 
times, you get ideas from data. Data allow you to test an idea, but you must 
also always be on the lookout for clues and hints which point in a different 
direction or which require you to deepen your analysis. Data analysis, there- 
fore, is akin to an open-ended dialogue. Never impose a story on the data, 
but equally do not expect the data to tell a story by themselves. As with an 
interview, if you do not ask relevant questions, you are unlikely to get 
interesting answers, but make sure that your questions are not worded as 
your own pet answers which you merely seek to confirm with a simple yes 
or no. 


An example may illustrate this point. Most people know from casual ob- 
servation that, in general, women outlive men. But how would you go about 
testing this proposition more broadly against empirical evidence? 
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One way to do this is to compare the life expectancies (at birth) of women 
and men across countries. The life expectancy at birth is a demographic 
indicator of mortality which shows the number of years a person would 
live on average if the present mortality rates for different age groups ap- 
plied to this person’s life. Note, however, that if, for example, life expect- 
ancy is 45 years (as is still the case in some African countries), this does 
not mean that most people die around 45 years of age. Rather, it principally 
indicates that infant mortality is still very high and that many people do not 
reach adulthood, thereby pulling the average life span downwards. 


So how can you test whether the life expectancy of women exceeds that of 
men? Statisticians do this through formal statistical testing which requires 
a knowledge of probability theory and statistical inference. To keep mat- 
ters simple, however, we shall use a straightforward graphical test which, 
in this case, does the trick equally well. To do this we shall make use of a 
scatter diagram which plots life expectancy of women (on the vertical axis) 
against that of men (on the horizontal axis) for each country in a sample of 
99 countries taken from the World Bank tables for 1990. Figure 11.1 shows 
the scatter diagram: each point in the plot corresponds to a particular coun- 
try and measures its female life expectancy on the vertical axis and its male 
life expectancy on the horizontal axis. 
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FIGURE 11.1 Female vs. male life expectancy: 99 countries (some circles represent 
more than one country with similar data) 


As you can see, there is quite a bit of variation in this plot. Life expectan- 
cies of men and women range from 40 to 44 years up to 74 to 81 years. 
There are significant differences, therefore, between countries. Further- 
more, as you would expect, the scatter of points slopes upwards within a 
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fairly narrow range. That should not surprise you. In countries where male 
life expectancy is relatively low (or high), female life expectancy will be 
relatively low (or high) as well. 


The plot features a 45° diagonal. Why is this? We drew this line for a very 
simple reason. As you can see from the graph, the diagonal traces the (hy- 
pothetical) loci of points where the life expectancies of women and men 
are equal. Thus, in general, if there were no differences between life ex- 
pectancies of women and men, the actually observed points would fall on 
the 45° line. If, however, life expectancy of women generally exceeds that 
of men, most points will be situated above the line. In contrast, the points 
would be situated mostly below the line if the life expectancy of women 
were generally below that of men. 


The scatter diagram shows clearly that for the great majority of countries 
the life expectancy of women exceeds that of men. This simple graph, 
therefore, presents strong evidence in favour of our initial proposition: 
namely, that, on average, women tend to outlive men. This is an example 
of testing an idea against the data. 


But the fact that we confirmed the initial proposition against the data does 
not mean that our analysis should stop here. While we were pleased that 
the data confirmed the hypothesis, this result did not come as a great sur- 
prise: we more or less expected it. But perhaps the data can also give us 
some new ideas? Take another look at Figure 11.1 and, before reading on, 
note down any features of the scatter diagram which particularly strike 
you. 


You undoubtedly noticed that there are five data points situated below the 
line. This means that, in these countries, women’s life expectancy is less 
than that of men: a feature which contrasts sharply with general world- 
wide patterns. Did you ponder which countries these might be? A policy 
researcher needs to be inquisitive and keen to follow up clues and hints 
thrown up by the data. Reading from left to right, the countries correspond- 
ing to these points below the line are, respectively, Bhutan, Bangladesh, 
Nepal, Pakistan and India. Does it surprise you then that there is quite a 
debate in the literature whether or not there is a gender bias against women 
in these countries (see, for example, Harriss, 1990)? It is not our intention 
here to discuss this debate. The main point worth noting, however, is that 
such debates are often prompted by unexpected patterns discovered in the 
data. These exceptions cry out for an explanation and, hence, push the 
analysis further. This way, you can get ideas from data. 


Did you further notice that, as you move your eyes from left to right along 
the graph, the scatter of points becomes more distanced from the 45° line? 
What this means is that the discrepancy between the life expectancies of 
women and men increases as life expectancy of both women and men in- 
creases. In fact, a bit more scrutiny of the data reveals that, in countries 
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where life expectancy is low (low-income countries), the difference in life 
expectancy between women and men is about 3 years, while, in contrast, 
this difference rises to about 6 years in countries where life expectancy is 
high (mainly, high-income countries). This is quite a significant pattern in 
the data which again throws up further questions. How to account for this 
rise in the difference in life expectancy between women and men as life 
expectancy increases? Are there specific risk factors (such as, for example, 
the greater risks of maternal mortality) which particularly affect women in 
poor countries? Or does women’s biological advantage only come fully 
into play with ageing and, hence, under social conditions where infant and 
adult mortality is low? These are questions which came to mind when we 
looked at this plot. As we are not specialists in health matters, however, the 
best research strategy would be to confront a specialist with these patterns 
and enlist possible explanations. 


In this example we used a simple device — a graph — to test ideas against 
data and to get ideas from data. In general, however, these two types of 
data analysis require different (but overlapping) skills. The former task is 
more akin to a court of law testing the hypothesis of not guilty according to 
the available evidence. The latter task is akin to detective work: looking 
for hints and clues which may guide you towards an explanation as to who 
committed a crime and how and why it was done. Confirmatory data analysis 
deals with the former; exploratory data analysis with the latter. Confirma- 
tory data analysis generally involves the use of heavy artillery — probabil- 
ity theory and statistical inference — and its mathematical threshold tends 
to be fairly high, given its extensive reliance on formulae used in formal 
statistical testing. Exploratory data analysis is more playful in nature and 
makes extensive use of graphical tools to look at data. 


Until recently, statistics texts dealt exclusively with confirmatory data analy- 
sis. The underlying idea was that scientific method should consist of test- 
ing ideas against data. Nowadays modern texts recognize the importance 
of getting ideas from data and devote attention to techniques which allow 
you to do this better. This greater emphasis on learning from data is un- 
doubtedly due to the ground-breaking work of Tukey (1977), the innova- 
tive American statistician and founder of exploratory data analysis. 


11.1.3 ADVOCACY, OBJECTIVITY AND CREDIBILITY 

Adopting a contrastive approach to data analysis is likely to render your 
empirical analysis more objective and fair, and so give it more credibility. 
This is not to say that it renders the analysis objective in some absolute 
sense. By its very nature this type of analysis will be relative, contextual, 
and historically specific. That is, it depends on the context in which your 
analysis inserts itself, the type of ideas which confront one another, and the 
type of evidence which is available at the time. What it means, however, is 
that you should set out your own argument by taking explicit account of 
the ideas and evidence put forward by others and substantiate your argu- 
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ment in the light of the overall evidence at hand. But why should you bother 
with rival ideas if, for example, your main purpose is advocacy of your 
particular point of view (as in Chapter 13)? 


One reason is that your argument will stand up better to scrutiny since it 
deals explicitly with rival ideas. In other words, it gives greater credibility 
to your analysis. Good advocacy should not be afraid to investigate its own 
stance critically. Indeed, if your point of view is so fragile that it cannot 
stand up to closer scrutiny, perhaps it is time to question your views. Fur- 
thermore, good empirical analysis should at times make you feel uneasy 
about the views you hold and prompt you to submit them to greater scru- 
tiny. If you never experienced such unease, perhaps you are too much in- 
volved in collecting evidence in favour of your own view, while develop- 
ing a blind spot for any loose ends thrown up in analysis or hiding them 
behind ad hoc justifications. 


To be for something does not mean that you should abandon judgement 
and cease to examine your beliefs critically. Policies and the public actions 
they inform, however well intended, invariably give rise to tensions and 
unforeseen outcomes. The failure to confront such tensions often results 
from the unwillingness to examine them critically for fear that this may 
reveal a lack of commitment. Yet, not uncommonly, the consequence of 
not confronting the undesirable outcomes of such policies may well be that 
the public action they support becomes discredited in the process. A will- 
ingness to maintain a critical stance, and the ability to cope with the inevi- 
table unease to which this may give rise from time to time, may well prove 
to be the best lifesaver for the public action you seek to advocate. 


In developing a critical attitude, fair contrastive analysis means that you 
take the ideas and evidence of opponents seriously and investigate them in 
their strongest formulations. It is all too easy to pick a weak variant of an 
opponent’s argument and then proceed to disprove, ridicule, and dismiss 
it. If you do so, you are merely attacking straw men, not real warriors. It 
may appear to render the task of analysis easier and may even give some 
short-term mileage in terms of getting rid of some of your critics, but, 
almost invariably, this happens at the expense of your longer-term cred- 
ibility as a researcher and policy analyst. 


11.2 THE FACTUALITY OF DATA 

11.2.1 DATA ARE THEORY INSPIRED 

Facts are not just given things; they are produced. That is, data are partial 
records which provide you with a selective visibility of a society and its 
economy (Hopwood, 1984, p. 170). This explains why Carr states that a 
fact is like a sack since it won’t stand up unless you put something in it 
(Carr, 1990, p. 11). Facts, therefore, are always theory inspired: it is theory 
which renders a piece of information relevant and, hence, accounts for it to 
be selected as a fact. Evidence never exists in isolation; it is selected with 
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reference to a particular idea which renders it relevant. That is the “some- 
thing’ which makes it stand up as a fact. 


Thus, data are always theory inspired. But this should not lead you to think 
that the substance of data is entirely reducible to theories and concepts or is 
purely subjective in nature. For example, national income accounts are a 
product of (Keynesian) macro-economic theory but this does not mean 
that the national income of India is constructed of thin air. To say that 
national accounts data are theory inspired means that theory determines 
which data are deemed to be relevant and informs the way these data are 
collected and structured into meaningful policy-relevant macro-aggregates 
such as the gross domestic product, national income, consumption, invest- 
ment, savings, and so on. 


In policy analysis, factual evidence is obtained from primary and second- 
ary data. The former are collected by the researcher, so it is the researcher 
who selects them, assesses their relevance, and shapes them as facts within 
analysis. This is a time-consuming process, but it allows the researcher 
greater flexibility in producing evidence for the research questions at hand. 
These primary data tend to be case and time specific because their collec- 
tion depends on the efforts of a single researcher (or a single team) operat- 
ing in a particular location during a specified period of time. Much de- 
pends on the skills of the researcher(s) in question and, given the case- and 
time-specific nature of the data collection and analysis, it is often impossi- 
ble to verify data or to correct inherent biases in data collection. Good 
primary data analysts are quite aware of this and make sure to explain the 
nature of their data and the procedures which went into their collection. 


Some primary data analysis involves fieldwork. Here, researchers gener- 
ally develop strong local knowledge, acquire a feel of the specificity of the 
location in question and, as producers of data, tend to be quite aware of the 
complexity, variability and uncertainty of social data. They are, therefore, 
less inclined to generalize too quickly and are more tentative in their con- 
clusions. Other primary data analysis is based on surveys carried out by 
the researchers themselves. For example, the above-mentioned study on 
rural households in Tanzania was based on survey data of 600 households 
in 20 villages. Such surveys are akin to, but generally smaller in size than, 
the large-scale surveys carried out by official institutions for the purpose 
of publishing secondary data. Survey analysts, unlike field workers, tend 
to be more distant from local circumstances and knowledge. 


Secondary data are produced by specialized (predominantly official) insti- 
tutions. Quite a lot of expertise goes into their production which is gener- 
ally repeated at regular intervals: national accounts, demographic data, trade 
data, social surveys, etc. These type of data are mostly aggregated using 
formal accounting frameworks which structure the data into predetermined 
categories. The production of such data, therefore, follows standard proce- 
dures and techniques which enhance their consistency over time as well as 
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their comparability between, for example, countries or regions. The use of 
aggregation and accounting structures extends the reach of these data be- 
yond local conditions. But, conversely, aggregation hides the internal vari- 
ability of the data within each category, and accounting practices often 
resolve conflicts among data through formal procedures. This often leads 
researchers who work with secondary data and who are mostly consumers 
(rather than producers) of these data, to see them as hard facts because a lot 
of the variability and uncertainty within such data has been carefully re- 
moved through aggregation and formal accounting practices. Not uncom- 
monly, therefore, these researchers (macro-economists, sociologists work- 
ing with survey data, or demographers) tend to be more affirmative in their 
views of reality and see their conclusions as being more objective than 
those of researchers involved in fieldwork. 


Streamlining the data through aggregation and accounting frameworks 
extends their reach at the expense of their flexibility in use. An example 
can best illustrate this point. Consider, once more, the case of life expect- 
ancy as an indicator of health. This is a very useful measure which allows 
you to make broad comparisons between countries, regions, town and coun- 
try, or between women and men. The ability to make such comparisons 
and to draw interesting inferences from them illustrates the power of sec- 
ondary data in extending the reach of analysis. But this attracts a cost as 
well. Most official published demographic surveys will give you life ex- 
pectancy by nation (or region, or town and countryside) and by sex, but 
few give you life expectancy by, for example, class or social status. So, if, 
say, your research leads you to question whether or not the mortality rates 
of urban manual wage labourers differ markedly from those of profession- 
als, you will find it very hard to investigate this hypothesis with published 
secondary data. The data simply are not structured this way and, hence, 
prevent you from investigating this question. The question is nevertheless 
potentially interesting since, for example, research in Britain has shown 
that, even in a developed industrialized country, mortality rates can differ 
markedly between social classes (Townsend and Davidson, 1988; White- 
head, 1988). 


As a researcher and policy analyst, therefore, you should always be aware 
that data never tell the full story. All data are partial records which yield 
selective visibility of a society or an economy. With secondary data, if this 
visibility is in line with the question you seek to explore, the data will help 
you do this. But, at times, the data remain mute because they are not struc- 
tured to answer the type of questions you are interested in. That does not 
mean that your question is meaningless, but it does make it harder to get 
the evidence you need to prove your point. You saw this in Chapter 1 where 
official data assisted in the administration of policies which had been set 
for some time, but did not help to answer questions needed once new poli- 
cies were under consideration. This explains why it is often important to 
combine the analysis of secondary data with targeted primary data collec- 
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tion which is aimed to address questions that the secondary data do not 
allow you to address. At times, however, policy analysts and researchers 
may be able to convince decision-makers to extend the reach of official 
data though the inclusion of new questions within existing surveys or by 
commissioning new surveys. In Tanzania, for example, after the adoption 
of structural adjustment policies in 1986, there was increased interest in 
what was happening in the informal sector in response to such policies, but 
it was hard to arrive at any firm conclusions for lack of data. This led to a 
first official survey of informal sector activities being undertaken in 1991. 


11.2.2 THE ‘DOMAIN OF THE FACTUAL’ 

In fact, the routine collection of official data forges a ‘domain of the fac- 
tual’ (Hopwood, 1984, p. 168) in that these data both reflect and, in turn, 
influence the emphases given in public debates (p. 169). For example, na- 
tional income accounting emerged to aid the apparatus of a constrained 
wartime economy in Britain, but after the war, the desirability of economic 
growth rendered GDP, once a means to an end, into a desirable end in 
itself, a process which subsequently shaped the nature of government poli- 
cies (Hopwood, 1984). In other words, once data enter the public domain 
they often assume a life of their own. They become the main indicators of 
success or failure and are used as such in public debates. The World Bank, 
for example, routinely classifies countries as high or low growth perform- 
ers and uses such categories to advocate specific policies which the latter 
should follow to achieve the success of the former. In similar fashion, with 
the rise of monetarism all eyes turned towards the growth of the money 
supply and of the public sector deficit, which became new indicators of 
success or failure. 


But official data also shape the way people perceive their position and role 
in society. Indeed, as the philosopher Hacking (1990, p. 3) pointed out: 


The systematic collection of data about people has affected the 
ways in which we conceive of a society, but also the ways in which 
we describe our neighbour. It has profoundly transformed what we 
choose to do, who we try to be, and what we think of ourselves. 
Marx read the minutiae of official statistics, the factory reports 
from the factory inspectorate and the like. One can ask: who had 
more effect on class consciousness, Marx or the authors of the 
official reports which created the classifications into which people 
came to recognize themselves? These are examples of questions 
about what we call ‘making up people’. 


(Hacking, 1990, p. 3) 
Official data, therefore, affect our consciousness in myriad ways. They 
enter public debates as indicators of failure or success and so acquire 


a quality of being ends in themselves, and they enter our lives by shaping 
how we see ourselves in relation to others. As Metz (1987, p. 224 and 
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p- 338) pointed out, for example, the development of unemployment sta- 
tistics in the nineteenth century gave rise to the concept of unemployment 
and became a subject of public debate in the later years of the century with 
the ascendancy of the working class movements. Similarly, the emergence 
of social statistics played a crucial role in the formation of public health 
policies. 

This quality of (socially sanctioned) greater factuality is specific to official 
secondary data and sets them apart from the ‘soft’ data of field workers. 
Policy analysts engaged in case-specific fieldwork often overlook this qual- 
ity of official secondary data, and, while often better informed on certain 
subjects, find themselves stonewalled by a barrage of hard facts when en- 
gaged in policy debates. As a policy analyst, you should never ignore the 
story told by secondary data even if you are convinced that your case- 
related data are more insightful and support a different story. In such cases, 
itis important to analyse carefully why the secondary data fail to reveal the 
specific concerns you seek to address and how your own data can help to 
bring out a different story. 


11.3 QUESTIONING SECONDARY DATA 

In Section 11.1, we have shown how evidence is used to choose between 
rival explanations. To do so, we used a few simple examples which asked 
you to assess a question or rival questions in the light of a piece of evi- 
dence. In actual research, however, it is you as the researcher who needs to 
formulate such questions and select the relevant piece of evidence. Alter- 
natively, you may come across a set of data (say, published statistics) and it 
is up to you to figure out whether these data help you to tackle the ques- 
tions you are interested in. Many researchers find this quite difficult to do. 
The main reason is that the data do not come precisely in the form you 
want them, and it may take some time to see whether they are of relevance 
to your concerns. In this section we shall use one example to teach some 
basic principles on how you might go about questioning secondary data. 


Suppose your interest is to analyse how the informal sector developed un- 
der structural adjustment in Tanzania. Indeed, since structural adjustment 
policies aim to give greater emphasis to private sector development (in- 
cluding informal sector activities), it is important to understand the spe- 
cific nature of the development of the informal sector under structural ad- 
justment, whether or not structural adjustment policies achieve their stated 
aims. You are looking for relevant data and you come across the informal 
sector survey of 1991. 


Take a look at Table 11.2. It displays a table taken from this survey. The 
table is organized in matrix form and involves cross-tabulations of the 
number of small-scale enterprises with respect to their location (urban/ 
rural), industrial sector and life span. Hence, the data in the table consist of 
counts, not measurements. 
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TABLE 1 
EOGRAPH 
Age (years) 
Geographical Upto! land<3 3and<5 Sand<10 10and<20 20o0rmore Notstated Total 
area/industry 
Dar es Salaam 
Agr, and fishing 3404 2658 1880 2067 1189 612 0 11 810 
Mining/quarry 0 0 0 0 0 0 0 0 
Manufacture 4579 = - 7340 7922 5062 3373 3180 0 31 456 
Construction 696 981 2654 2398 3128 905 0 10 762 
Trade/rest/hot 31863 43333 27927 27045 7982 3544 347,142 041 
‘Transport 382 595 462 525 197 153 0 2314 
C.andP. services 1076 2489 2005 4054 2332 803 0 12759 
Total 42000 57396 42850 41151 18 201 9197 347 211 142 
Other urban 
Agr. and fishing 5905 13782 12159 6040 4957 9822 0 52 665 
Mining/quarry 654-2173 3243 4773 2344 0 0 13 187 
Manufacture 8978 13 041 9599 10 250 7149 4434 104 53 555 
Construction 682-2006 43380 5125 4053 1842 48 18 136 
Trade/rest/hot 69943 77147 58099 33989 14 466 9855 2645 266 144 
Transport 614 1206 2152 1133 368 84 0 5557 
C. and P, services 1838 9062 3617 4614 4356 3489 45 27021 
Total 88614 118417 93249 65924 37 693 29 526 2842 436 265 
ees 
Rural 
Agr. and fishing 13 179 7440 13852 22387 11 330 9446 0 77 634 
Mining/quarry 95 1517 190 1488 662 0 0 3952 
Manufacture 20950 54691 62001 83943 59 368 69 572 4004 354.529 
Construction 5943 14.302 18 368 19 432 13 836 15 258 459 87598 
Trade/resvhot 93467 132959 114591 95 206 53.848 30941 4718 525 730 
‘Transport 10668 9129 11877 6201 2237 743 653 41 508 
C.andP. services 544) ‘11 247 7169 14727 13 960 9344 1297 63 185 
Total 149743 231285 228048 243 384 155 241 135 304 11131 1154 136 
aes 
Total 
Agr. and fishing 22488 23880 27891 30 494 17 476 19 880 0 142 109 
Mining/quarry 749 3690 3433 6261 3006 0 0 17139 
Manufacture 34507 75072 79522 99255 69 890 77 186 4108 439 540 
Construction 7321 17289 «25402 =. 26.955 21017 18 005 507 116 496 
Trade/rest/hot 195273 253439 200617 156 240 76 296 44 340 710 933 915 
Transport 11664 10930 14.49] 7859 2802 980 653 49379 
C.andP. services 8355 22798 12791 23395 20 648 13 636 1342-102. 965 
Total 280357 407098 364147 350459 211135 174 027 14320 1 801 543 
Source: The Planning Commission and The Ministry of Labour and Youth Development (1991) Tanzania: The 
Informal Sector, pp. 2-5. 
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You now have to decide whether this table might help you to pursue spe- 
cific questions concerning the possible effects of structural adjustment 
policies (initiated since 1986) on informal sector development. This is a 
typical research situation for quantitative data analysts. 


However, in our experience, many researchers find it difficult to know 
what to do next. The reason why they find it difficult is that nobody tells 
them which question to address or what technique to use. It is up to them to 
assess the relevance of these data. Some mainly stare at the table, and, 
possibly, end up by calculating some percentages or growth rates in a rather 
haphazard manner. They may decide the table is an important piece of 
evidence, but do not really know what to do with it, and end up reproduc- 
ing the table in their research report without the data ever being put to 
work. We call this a strategy of hoarding data: the research report may look 
impressive because it is full of tables, but it is likely to be thin in analysis. 


So how should you go about using this table more systematically and crea- 
tively? There are no hard and fast rules how to do this, but some simple 
principles can be formulated which may help your approach to data explo- 
ration. Let us use this example to illustrate some of these principles. 


The first principle is very simple, yet unfortunately often forgotten. It con- 
sists of taking time to look carefully at the structure of the table, its various 
categories, and the way they are cross-tabulated. Having done this, reflect 
upon and note down any important conclusions you can draw about the 
limitations inherent in the data in terms of what they can possibly tell you 
of relevance to the questions you are interested in, in this case possible 
effects of structural adjustment, which might show themselves in terms of 
changes after 1986. Do this now for a few minutes and then read on. 


In this case, we noted down that the survey covered enterprises existing in 
1991 and, hence, information about the past was obtained through recall 
(that is, asking people when they started their enterprises). We concluded 
that, since mortality rates of informal sector enterprises at different periods 
in time are unknown, it is no use trying to calculate growth rates in the 
number of enterprises over time. Yet many people make this error. It is 
indeed tempting to try to use the table to show that the number of informal 
enterprises has been growing over time, but this would be wrong because 
you have no information how many enterprises actually existed in the past 
but ceased to exist before 1991. There are no a priori reasons, however, to 
believe that mortality rates of enterprises would differ markedly across 
either sectors or the rural/urban divide. Consequently, with care, the data 
might be used to explore structural changes in the spread of the informal 
sector across activities before and after the implementation of structural 
adjustment policies and across the rural/urban divide. The reason why it is 
possible to compare changes before and after structural adjustment is that, 
by 1991, structural adjustment policies initiated in 1986 had been in place 
for about 5 years; thus the breakdown of the table along columns allows 


253 


THINKING WITH DATA 


you to look at the proportion of different types of enterprises which are less 
than 5 years old compared with those of 5 years or more. 


Second, after studying the structure of the table, merely staring at the data 
does not help. To engage in a meaningful dialogue with data it is necessary 
to fire questions at the data. Each question approaches the data from a 
different angle. Some avenues may prove to be dead-end streets, while 
others yield valuable insights. Here, one question which sprang to mind 
was: is there an inherent bias in favour of trade, as opposed to production 
or services, among informal sector activities under structural adjustment? 
What made us think of this question was that, if structural adjustment mainly 
fuelled informal sector trading activities, there is a limit to the dynamism 
this sector could display in bringing about economic recovery and growth 
in Tanzania. Much depends also on the role the informal sector plays in 
reshaping production. So, as a preliminary question we decided to check 
whether there was a bias in favour of trade. 


This is only one of several questions which can be raised about the table. 
Each question gives you a different ‘handle’ on the data. Each handle al- 
lows you to look at your data and gain specific insights. This is what selec- 
tive visibility is all about: the question is the handle by which you make 
things visible. Exploratory analysis consists of trying out many different 
questions, exploring each of them in turn and following up those questions 
which appear more promising. Here we shall only pursue this particular 
question, but keep in mind that in actual research it is important to ap- 
proach the data from different angles. 


Third, it is useful to collapse the table so that it emphasizes the question we 
want to pursue. The point is not to carry around any extra baggage in terms 
of data overload. In this case, the table can be simplified along three di- 
chotomous categories: trade/non-trade, enterprises with a lifetime of less 
than 5 years versus those of 5 years or more (since structural adjustment 
policies started in 1986 and the survey was held in 1991), and the rural/ 
urban divide. 


Tables 11.3 and 11.4 summarize respectively the urban and rural picture. 
Notice that what we have called ‘Trade’ in Tables 11.3 and 11.4 corre- 
sponds to the rows labelled “Trade/rest/hot’ in Table 11.2. We might have 
preferred figures on trading enterprises separate from restaurants and ho- 
tels, but as noted above, secondary data are generally not in precisely the 
forms one would prefer. We will have to assume that any change in the 
relative numbers of trading enterprises will be reflected in the change re- 
corded for the ‘Trade/rest/hot’ category, taking care to note this assump- 
tion and to check it later if possible. These tables do away with unneces- 
sary clutter and focus your attention on the question under review. This 
allows you to get a better grip on the data. Take a good look at both tables. 
You will note that the information is now structured in line with our spe- 
cific question. 
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TABLE 11.3) URBAN INFORMAL SECTOR ENTERPRISES (NUMBERS OF ENTERPRISES) 


Age of enterprise Sector Total 
Trade Non-trade 

<5 years 308 312 134 214 442 526 

>5 years 96 881 104 811 201 692 

Total 405 193 239 025 644 218 


TABLE 11.4 RURAL INFORMAL SECTOR ENTERPRISES (NUMBERS OF ENTERPRISES) 


Age of enterprise Sector Total 
Trade Non-trade 

<5 years 341 017 268 059 609 076 

>5 years 179 995 353 934 533 929 

Total 521012 621 993 1 143 005 


Fourth, the question now arises whether to continue working with the raw 
data (i.e. data in their original form) or to look for a suitable transformation 
of the data. Here the raw data are counts. The question now is whether we 
should continue to work with counts or perhaps switch to, say, percentage 
shares? Many ignore this question or, worse still, choose a particular trans- 
formation of the data without giving it much thought. This is highly unfor- 
tunate. Data transformations can either erect a smoke screen which pre- 
vents you from seeing any patterns in the data, or greatly enhance your 
analysis. It is worth giving this question some careful consideration. In this 
case, we decided to try out percentages as the appropriate data transforma- 
tion since we are mainly interested in looking at structural changes. Table 
11.5 gives the resulting cross-tabulation of percentage shares. 


TABLE 11.5 PERCENTAGES OF TRADING ENTERPRISES IN TOTAL NUMBER OF 


ENTERPRISES 


Age of enterprise Location 

Urban (%) Rural (%) 
<5 years 70 56 
>5 years 48 34 


Did you notice that we have now compressed all the relevant information 
into a single table? The structure of the data in relation to our original 
question has now become much clearer. To take a better look at these data, 
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however, it is useful to add an extra row and column to record the differ- 
ences in the percentages. This simple technique is illustrated in Table 11.6. 
The number in the bottom right corner records the difference of differ- 
ences, which, in statistical parlance measures the interaction effect between 
both categorical independent variables (policy regime and urban/rural lo- 
cation) on the categorical dependent variable (trade/non-trade). 


TABLE 11.6 PERCENTA TRADI 4 S IN TOTAL NU 
ENTERP ) N DIFFEREN’ 


Age of enterprise i Row differences (%) 
Rural (%) 


This convenient set-up allows us to draw the following three conclusions: 


1 The proportion of trading enterprises in the total number of enterprises 
is higher in the urban as compared with rural areas both before (48% vs. 
34%) and after (70% vs. 56%) the start of structural adjustment poli- 
cies. 

2 After structural adjustment, the proportion of trading enterprises rose 
from 48% to 70% for urban areas, and from 34% to 56% in rural areas. 


3 The change in favour of trading activities with structural adjustment is 
equally strong in urban and rural areas: the percentage shares increase 
by 22% in both cases. There does not appear, therefore, to be any urban 
(or rural) bias in this change towards greater emphasis on trading ac- 
tivities. 

These conclusions may help us in evaluating alternative notions as to the 
effect of structural adjustment on informal sector activities. We shall not 
pursue this analysis further here. The purpose of this example was to show 
how you can make data come alive if you care to fire a question at them 
and follow it up systematically by collapsing your data in line with the 
question, choosing an appropriate transformation, and selecting an appro- 
priate technique. For didactic reasons we have kept the technical sophisti- 
cation minimal: the calculation of percentages, addition and subtraction is 
all you need. More skilled data analysts might have used odds instead, or, 
better still, the logarithms of the odds (logits). These tools are more power- 
ful and may yield insights where percentages fail to do so. Fortunately, in 
this case, percentages tell the same story. 


Did you notice that, while our first and second conclusions might have 
been directly glimpsed (with difficulty) from the original table, the pattern 
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revealed in the third conclusion would be very hard to spot if we had not 
gone about our analysis systematically? But this was to us by far the most 
interesting feature in the data. Why? Well, with our background knowl- 
edge of Tanzania, we more or less expected the first and second patterns to 
prevail, but the third conclusion came somewhat as a surprise to us. Our 
earlier hunch was that the change towards trade was more town centred, 
The data tell a different story. But this is what a dialogue with data is all 
about. 


11.4 MATCHING CONCEPTUALIZATION WITH SKILLS IN 
QUESTIONING DATA 

Let us now try to draw up some lessons about the type of problems you are 
likely to encounter when dealing with quantitative data in applied policy 
research. 


Data do not tell you anything unless you fire questions at them. In fact, the 
search for data is itself driven by the type of questions you seek to explore. 
Conceptualizing your research question, therefore, is essential to deter- 
mine your approach to the data. But good conceptualization is akin to an 
opening in a chess game: it sets the stage for the game but does not prede- 
termine its final outcome. 


Research almost invariably implies that you approach data from different 
angles. Do not be afraid to try out different things. Some may turn out to be 
dead-end streets, but others will yield valuable insights. But there never 
are uninteresting results. Even if some result is unexpected or disappoint- 
ing, it is worth your while to follow up the question of why the evidence 
fails to back up an idea you held. This may point you towards a rival idea 
which you might otherwise not have taken into consideration. 


You approach data from different angles because you want to investigate 
rival notions in the light of the evidence each brings to bear on the problem 
at hand. Through contrastive inference you seek to arrive at that notion 
which appears more plausible in the light of the overall evidence. Your 
interest is not just to test a particular idea in isolation — or what Miller 
called a lonely encounter of a hypothesis with evidence (Miller, 1987, 
p. 173). What matters is to compare ideas. The implications are that you do 
not merely accept an idea because some evidence points towards it, be- 
cause it may support other ideas equally well, and you pay particular atten- 
tion to evidence which goes counter to an idea, because it may point to- 
wards a superior notion. 


In policy analysis, you may make extensive use of secondary data relevant 
to your research question. It is important, therefore, to reflect carefully on 
the limitations inherent in the data. This is not just a question of checking 
the quality of the data. What matters as well is to ascertain the type of 
questions data allow you to answer and which questions cannot possibly 


257 


THINKING WITH DATA 


be tackled with the data at hand. This point matters equally when you col- 
lect your own data (for example, through survey research). The design of a 
survey is predicated by the type of questions you seek to answer. 


Analysing data in relation to a specific question almost invariably implies 
further data manipulations. In the example above, we collapsed the table to 
be able to get a better grip on the data. In the process, however, new vari- 
ables emerged. For example, we used the age of enterprises to define a 
proxy variable for policy regime depending on whether the enterprise was 
started before or after the adoption of structural adjustment policies. In 
other cases, you might use an index of average nominal wages jointly with 
the consumer price index to construct an index of the evolution of real 
wages, or compute the real exchange rate using a set of domestic and for- 
eign price indices and the nominal exchange rate, etc. Data manipulations 
are the ‘bread and butter’ of good applied research. Good conceptualization 
of research often reveals itself first by the quality of data manipulations 
and, conversely, careless thinking often shows itself in the way data are 
organized or manipulated. 


The question you seek to answer, the type of data at your disposal, and the 
way you manipulate and transform them, jointly set the stage for the kind 
of techniques best suited to address the question. This is not a one-way 
street. Often data are manipulated in certain ways so as to make them ame- 
nable to the use of a particular technique. Nor is it a one-shot effort. In the 
example above, we could have used odds ratios or logits, but percentages 
did the trick just as well and require less skill. We could also have used the 
raw data in the context of a contingency table, but that did not give us 
additional mileage. In general, trying out different techniques using the 
same data to answer the same question is a good thing. This way you avoid 
turning a technique into a fetish: a tool to impress rather than to gain insights. 


Numerical skills matter provided they are not used in a mechanical way. 
They should be used in combination with, and not as a substitute for, clear 
conceptualization of the problem. This is why we called this chapter “Think- 
ing with quantitative data’. Numerical skill in itself does not provide magi- 
cal answers to problems. Good conceptualization of a problem determines 
whether numerical skills are used appropriately or foolishly. In the exam- 
ple above, we could have used the table to calculate apparent trends in the 
growth of the numbers of informal sector enterprises in different sectors. 
These results would have been meaningless, however, since the data are 
prepared at a given point of time and, hence, do not reveal how many en- 
terprises ceased to exist in the past. The mechanical application of numeri- 
cal skills, therefore, can be (and often is) quite misleading. But, conversely, 
to undertake good data analysis requires you to be numerically literate. 
Few patterns in the data can be seen by merely staring at them. It is through 
data manipulations, data transformations and the calculation of numerical 
summaries or the use of graphs that conceptualization interacts with nu- 
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merical skill to bring out patterns in data which either test or generate ideas. 
There is no easy shortcut, but fortunately much mileage can be obtained 
with relatively simple techniques and extensive practice. 


Finally, good data analysis should be fun, although it may involve hard 
work. There is something very exciting about ‘finding out’. This is why 
data analysis should never just consist of testing old ideas with new data, 
but new data should also generate new ideas (Heckman, 1992, p. 884). Do 
not hammer your data into submission in a mindless exercise of number 
crunching. Nor turn your back to them in an attack of data phobia. It is 
worth learning to converse with quantitative data in a genuine theory-driven 
dialogue. 


259 


—— es 


= COE ar mene tell wails Rade Ce 0: pan Ered eS 
betaine of wesSgeinw <oaeii tinceroe] oat Sci pie co @ 
< * SMEYY Steals Se epee caer 
jet hos yu eles os a eal eeeiene 
a ofS eid? “el pee) ok pee ee 
. wee ae aes Be gees Woes 
3 ot) Ree a EPL geeerborety 
x iii bois ects 
77 14s] ak > Gee 
= De eee) ‘3 


12 LYSING INSTITUTION 


GEOFF JONES 


CCOUNTS 


In developing any programme of public action, the institutional framework 
within which it is conceived will have an important impact. Chris Roche in 
Chapter 8 suggests how you might find out a range of things about an 
organization or institution from the representations and information it and 
its members publish or distribute and the reasons why these might take 
particular forms. His approach is fairly generalized but offers a model or 
series of perspectives which can alert the researcher to a whole range of 
important aspects of an organization’s values, processes, actions and 
effects. 


This chapter examines how organizations and institutions present their work 
and project their image through accounts — what we can learn about an 
organization from its accounting ‘footprints’. In many senses it is a re- 
stricted perspective but paradoxically, it may be a more fundamental or 
essential one — it can perhaps reveal what organizations and institutions 
‘really’ do. According to this perspective, what they do is channel resources 
from some groups of people to others, possibly adding some ‘value’ to 
them in the process. Accounting is the means by which these flows and 
accumulations of resources are represented in monetary terms. Accounting 
therefore has important but complex relationships with the discourse of 
economics (Hopwood, 1992). In the context of development management, 
as elsewhere, an emphasis on accounting as a prime means of promoting 
accountability and control has been a significant part of recent public man- 
agement reform in support of privatization, contracting, income-genera- 
tion, cost-recovery or donor accountability (Flynn and Strehl, 1996; Hood, 
1995; Mackintosh, 1995). 


In this chapter, we consider some of the ways in which analysing accounts 
can help us to ‘find out fast’ about some of the fundamental aspects of an 
organization, for example, what its financial resources are, where it gets 
them from and how it uses them. However this chapter is not meant to be 
an all-purpose accounting primer. It concentrates on those types of accounts 
and practices of accounting which you are most likely to come up against 
in development policy and public action: what I have called ‘institutional 
accounts’. These consist of the accounts of mostly not-for-profit entities 
such as non-governmental organizations (NGOs), government agencies and 
departments, both domestic and international. But nowadays these organi- 
zations commonly engage in some form of profit-making or revenue-earn- 
ing activities, and we will consider some of the accounting aspects of these. 


“Giving accounts’ is itself an institution practised by all kinds of organiza- 
tions, so hard and fast definitions about what is and is not an institutional 
account may be difficult to establish unambiguously. But the term ‘institu- 
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tion’ is used in this chapter to mean organizations or accounting entities 
(such as projects) which have certain established and durable patterns of 
activity, norms and practices (which include accounting). Thus a multi- 
national corporation like Shell or BP can be regarded as an institution, as 
well as a project team or government department (but the accounts of multi- 
national commercial organizations are not considered here). 


I hope that this chapter will give you some ideas about how to analyse and 
interpret institutional accounts, for example how to: 


* calculate some simple ratios, such as the proportion of total expenditure 
or assets that are accounted for by particular classes of assets or ex- 
penditure, such as debtors and creditors; 


* compare one year’s figures with another; 
¢ decide which of the figures in the accounts are worth investigating; and 
© judge what kinds of question it is reasonable to ask. 


We won’t consider the intricacies of financial ratio analysis or debt struc- 
tures usually associated with analysis of commercial accounts. If you feel 
you need some help with these, consult one of the many accessible ac- 
counting manuals available such as Mott (1993) or Elliot and Elliot (1996). 


We begin in Sections 12.1 and 12.2 with some fundamental questions and 
distinctions to make when analysing institutional accounts. In Section 12.3 
we investigate examples of accounts from different types of organization 
to see what sorts of things they can tell us. Section 12.4 considers the means 
by which we can have some confidence that what accounts seem to be 
telling us is reliable and valid. Section 12.5 identifies some of the problems 
that accounting conventions and practices present and we conclude in Sec- 
tion 12.6 with a brief look at how some of these problems are being tack- 
led. 


12.1 WHY ANALYSE INSTITUTIONAL ACCOUNTS? 

Like any question of this kind, the answer is: ‘it depends’. It is very diffi- 
cult to envisage a generalized range of enquiries which would cover all 
possible motives. And there is an obvious relationship to the purposes for 
which the organizations themselves draw up and publish accounts. In the 
search for what may be ‘hidden’ in accounts we should not overlook the 
ostensible reasons why they are prepared or published. One of these rea- 
sons — and one of the important effects of accounting — is to ‘make visible’ 
certain aspects of an organization’s operations and their impact. But at the 
same time, we should be wary of assuming that accounting representations 
reflect ‘the truth’ about an organization or its underlying ‘reality’. We will 
return to this issue later. 


Like other data accounting data can only be converted into knowledge 
by firing questions at them (Chapter 11), so this section should give you 
some idea about the kinds of question it is reasonable to ask, even though 
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accounting cannot give you all the answers and will inevitably leave some 
loose ends which may need to be followed up by further, different types of 
investigation. 


Which questions to ask will depend on the type of action envisaged as 
resulting from the investigation. As noted in Chapter 8, there is a big dif- 
ference between organizational assessment aimed at change within the or- 
ganization and seeking understanding from an organization’s ‘footprints’ 
in order to inform public action. In different ways, accounts can play a part 
in both cases. 


The kinds of question which can usefully be asked in different circum- 
stances could be, for example: 


* questions about how accounting/financial management is done; 

* questions about what the activities of an entity are, how it is funded, 
what it says about itself, etc.; 

* standard questions about financial performance; 

* question about performance which relate to the organization’s achieve- 


ment of its aims and objectives (these are unlikely to be purely financial 
so the contribution that accounts can make will vary from case to case). 


In the examples that follow, I will suggest which of these different types of 
question are more suitable to ask of the information provided. 


From a development management perspective, a more detailed set of pur- 

poses for analysing accounts could be to: 

* get evidence of good management or good stewardship of the resources 
under an organization’s control; 

* evaluate an organization’s ability to undertake projects or participate in 
a partnership or contract of some kind; 

* discover the sources of an organization’s income, the uses to which it is 
put and with what success; 

* assess an organization’s ability to generate its own income rather than 
rely on external funding or borrowing; 

* understand the financial strategy of a funding or commissioning organi- 
zation, government department or service provider; 

* find out what kinds of activities the organization is involved with and 
where; 

* find out whether the organization conforms to recognized international 
accounting conventions and/or whether it makes any disclosures about 
its social or environmental policies. 


The way in which these purposes are related to the form that accounts take 
and how suitable or otherwise the accounts are for these purposes needs 
some comment, especially in relation to our main concern here, which is 
not-for-profit NGOs and governmental institutions. 
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Commercial accounting, at least in the Anglo-American tradition, places 
great emphasis on the role of the shareholders, the owners of the business 
(who may be pension funds or an investment trust as well as individuals). 
Thus most accounts and international accounting regulations (which are 
based on the Anglo-American view of accounting) pay great attention to 
identifying what the business has generated in terms of ‘shareholder value’. 
In this tradition, the balance sheet (the statement of assets and liabilities, 
which, according to accounting conventions, must equal one another) is 
thus the most important accounting statement, and much financial analysis 
is based on comparing two or more balance sheets for the same accounting 
entity at different times. The ‘entity’, such as the firm, is the ‘thing’ which 
is accounted for. Examples of other accounting entities are funds, projects, 
and of course NGOs or governments. The concept of the ‘accounting en- 
tity’ is important and you need to realize that the ‘entity’ need not be an 
organization. 


Funds or projects are probably more significant in development manage- 
ment than they are even in other not-for-profit situations. The object of the 
accounting treatment of them is usually to show how the total expenditure 
relating to a particular objective or project can be broken down into sala- 
ries and wages, capital expenditure, equipment, publicity, educational pro- 
grammes, fees to third parties and so on, and how it has been financed — 
the total income from grants, voluntary aid, donations, sales etc. 


In the case of projects, the final accounts (covering several years and show- 
ing how the total input by governments and other donors have been spent 
and including a report on the benefits of the project) may take several years 
to compile. This can be long after the project has been completed or its 
outputs have been brought into operational use. 


Funds are rather different in that they retain the conventional accounting 
focus on the year-by-year stewardship — how much funds have grown or 
by how much they are depleted, how and where the accumulated balances 
are held (e.g. short-term bank deposits or longer term securities). 


From a development management perspective, it is important to recognize 
that what counts as ‘good accounting’ for entities like the World Bank or 
the IMF is largely based on the Anglo-American tradition, even when this 
is imposed in Africa or Russia and Eastern Europe. But it has frequently 
been argued that the implied emphasis on balance sheets is not as impor- 
tant to not-for-profit organizations or projects because it is no part of their 
core function or objectives to increase their net worth, which is what bal- 
ance sheets measure. 


The other main international commercial accounting tradition, the Franco- 
German, places a higher emphasis than the Anglo-American on other 
stakeholders such as creditors (especially lenders and tax authorities). Thus 
in this tradition, there is a greater emphasis on the income statement (the 
record of income and expenditure showing how current revenues were 
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obtained and how they were spent to arrive at a gross surplus or profit 
which can then be appropriated to various uses, such as expenditure on 
capital assets). There is also a greater emphasis on the associated conserva- 
tism of ‘booking’ expenses immediately but recognizing profits only when 
they have been realized (but this is only a relative emphasis, since con- 
servatism, or the ‘prudence’ principle is part of both traditions). 


Accounting in institutions such as NGOs and governments has historically 
drawn on commercial accounting principles and practices but has devel- 
oped a set of conventions of its own based around the significance of cash- 
based accounting. Because there is an emphasis on accountability for funds 
raised through taxation, government grant (essentially the same thing) or 
fund-raising, institutions emphasize their stewardship function (i.e. how 
well they have ‘looked after’ the funds they have been entrusted with). 
This stewardship function can include both what has been done with the 
current year’s income and how the capital or fixed assets of the entity have 
grown or diminished over the years, to what extent the organization has 
been dependent on external grants or loans and so on. 


This idea of stewardship also includes what has been ‘achieved’ with the 
available funds, or at least how and where they were spent. Throughout the 
world, this perspective has led to a focus on ‘value for money’ within not- 
for-profit organizations (e.g. Hood, 1995). And even the act of publishing 
accounts can demonstrate a certain organizational capability and ‘respon- 
sible’ attitude or accountability. 


The stewardship function is important in commercial accounting also, but 
for the reasons outlined above is more associated with stewardship of the 
explicit assets owned by the entity (as disclosed in the balance sheet) than 
with the use made of funds raised from customers or lenders. For example, 
many items of income and expenditure, such as proceeds from asset sales, 
are not included in a company’s profit and loss account on the grounds that 
they are not part of its ‘normal’ operations. Many of these conventions 
have influenced accounting practices in the not-for-profit sector. 


Thus insofar as public management reforms have increased the saliency of 
accounts, the conventions which have been most drawn on are those in the 
Anglo-American or ‘balance sheet’ tradition, even though there are clear 
consequences for the ‘revenue’ accounts (the day-to-day income and ex- 
penditure). For example, the introduction of depreciation accounting (ac- 
counting for the extent to which assets wear out in use) or capital charges 
(a surrogate for depreciation) in the accounts of local and health authori- 
ties in the UK focused attention initially on what the assets were ‘worth’ — 
even though one of the main reasons for introducing the changes was to 
focus the decisions managers made about the use of assets (Jones, 1996). 


We might therefore conclude that accounts can be useful for assessing stew- 
ardship and management capability, and for not-for-profit organizations it 
is the income statement which is likely to be of more interest than the 
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balance sheet. And close analysis of income and expenditure accounts can 
shed light on many of the other purposes of analysis suggested above. Gov- 
ernmental balance sheets are of little use for analysing programme expendi- 
ture. Balance sheets are, however, useful when looking at inter-organiza- 
tional relationships and partnerships, indebtedness and capital projects. 


12.2 WHAT ARE INSTITUTIONAL ACCOUNTS? 


I have already referred to some of the main components of accounts: the 
balance sheet and the income statement. But there is an even more funda- 
mental distinction to be made first, and that is between accounts which are 
published and accounts which are kept either as the underlying records 
from which the published accounts can be constructed (the financial ac- 
counts), or as internal management accounts, such as budgets, unit cost 
comparisons and investment appraisals, which are used primarily for mana- 
gerial decision-making. (Some of these management accounts are now fre- 
quently available in the public domain as part of public accountability ini- 
tiatives.) Thus the published financial accounts are produced primarily for 
external reporting reasons rather than to help managers run the organiza- 
tion — though some of the underlying records, such as debtors, creditors, 
stock levels, grant income and loan repayments form an important part of 
cash flow and working capital calculations which managers need. 


The rules which require commercial accounts to be published are covered 
by international accounting standards set by bodies with international ju- 
risdictions or, in the case of multinational corporations, are probably cov- 
ered by the requirements of the accounting regulations in force in the coun- 
tries in which the headquarters of the corporation are located or where the 
company is registered. As noted above, the reasons why commercial ac- 
counts are published will differ significantly from those applying to not- 
for-profit organizations, although they are likely to find themselves subject 
to the same set of rules. There are some exceptions — small organizations 
(measured by turnover/ gross income or total expenditure), and some trusts 
or NGOs limited by guarantee, or government trading organizations, may 
be exempt from some or all the regulations. However, the requirements of 
public accountability — arising through reliance on government taxation or 
grant, donors or fund-raisers — may be more onerous than those imposed by 
commercial accounting standards. 


With such a range of possible requirements, it is impossible to say exactly 
what circumstances in what countries will require what kinds of accounts 
to be published and under what accounting regulations. Nevertheless, there 
are a range of broad principles, known as Generally Accepted Accounting 
Principles, or GAAP, which are likely to be applied. These mostly consist 
of the output from a range of regulatory and professional bodies (domi- 
nated by US accounting practice and practitioners), but could also include 
vague influences such as ‘common business usage’ (AICPA, 1975). The 
most important of these principles is that the financial statements of a com- 
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pany give a ‘true and fair view’ of the state of affairs of the company and 
the result of its operations, ‘in conformity with generally accepted account- 
ing principles’, consistently applied. 


A further complication is what can be meant by the term ‘publish’. This 
can mean a full audited set of widely circulated accounts adopted by a vote 
of members at a public meeting. Alternatively, it can mean a few sets of 
typescript accounts given by an NGO to its sponsoring government depart- 
ment, where it may be hard to establish their existence, let alone get hold 
of a copy. The principle of public inspection of all accounts filed in a pub- 
lic registry and available to anyone (usually on payment of a search fee) is 
not universal, and even if such a system is in operation, it is unlikely that 
all the accounts for all the years you might want to inspect are available, 
especially in developing countries. Fears about commercial confidential- 
ity or the simple inefficiency of the regulators are frequent reasons why 
accounts are unavailable. Even in the UK, accounts for large companies 
may remain unfiled in the public registry for several years before any ac- 
tion is taken against them (even though, for most companies and some 
charities, there are strict laws requiring this to be done). 


In the governmental and NGO sector, where the reasons for filing accounts 
are more to do with public accountability than reassuring shareholders, 
stock markets or creditors, the public availability of accounts may be even 
more haphazard. Government accounts, in particular, at programme, project 
or departmental level, are unlikely to be covered by disclosure laws. Small 
NGOs may not have the resources to publish accounts, or they may be 
governed by different regulators, such as the Charities Commissioners in 
the UK, who impose different reporting requirements (for information on 
these see, e.g. Community Matters, 1997). A further complication may be 
the language in which the accounts are prepared or certain differences be- 
tween national and international accounting practice. Although English is 
the most widely used language of international accountancy, in some set- 
tings, for example in some Islamic countries, China or certain parts of Af- 
rica such as Ethiopia, accounts may need to be specially translated for an 
international audience. 


Accounts normally take a certain well-defined form, whether for a com- 
mercial organization or an NGO or government trading fund. There is the 
Directors’ or Trustees’ Report, introducing the accounts and commenting 
on the significant achievements of the year, the Auditor’s Report, the In- 
come and Expenditure/Profit and Loss Account, the Balance Sheet and 
Notes to the Accounts. The whole set of documents is usually referred to as 
the Annual Report of the organization. 


In the case of NGOs and governmental accounts, there may be separate 
accounts for funds or endowments with specific objectives, although the 
balances vested in such funds will be included in the consolidated balance 
sheet. More generalized governmental accounts such as national income 
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accounts are not normally recognized as financial accounts, being concerned 
with measuring macro-economic variables such as gross domestic product 
(GDP) even though company accounting information and governmental 
programme expenditure is used to compile them. 


To explain some of these ideas more fully, we are going to consider a range 
of actual accounts to see what they can tell us. 


12.3 SOME DIFFERENT TYPES OF ACCOUNTS 
12.3.1 A SMALL NGO ACCOUNT 


Let us consider Example 12.1, an account from a small NGO. Thinking 
back to the kinds of question these accounts might be able to answer rea- 
sonably well (Section 1), I suggest that they might shed useful light on how 
accounting and financial management is done in the organization, how it is 
funded and how good is its financial performance. These accounts are use- 
ful both for organizational assessment and to inform public action, although 
they give little information about exactly what is being done, how and why. 
Thus if you were a local taxpayer, concerned to know how the local au- 
thority’s money was being ‘looked after’ by this organization, these ac- 
counts could provide some useful starting points. 


There are a number of things we can note from Example 12.1. Firstly, these 
accounts are not the annual accounts for the organization but a monthly 
report prepared for a specific purpose such as a meeting of the management 
committee. They contain only one month’s transactions. The turnover 
of this organization looks as though it will be well over £10000 per year 
(comprising what looks like a start-up grant plus recurring income of around 
£2500 per month) but under £100 000. It thus will be required under the 
UK Charities (Accounts and Reports) Regulations 1995 to produce and 
submit an annual report and accounts and an annual return to the Charities 
Commissioners and undergo an independent audit. 


However, it does not have to produce its accounts using the accruals con- 
vention — where accounts include items relating to the accounting period 
which have not yet been received or paid —as it has no fixed assets or stock. 
Accounts based only on actual cash received and paid are called, for obvi- 
ous reasons, receipts and payments accounts. In Example 12.1, the organi- 
zation has chosen to accrue for certain items which it has paid for but which 
do not relate to the period of account (rent, rates, insurance), which it owes 
but has not yet paid (mother tongue teaching), or which it has received but 
which relates to a subsequent accounting period (*LBH grant recieved in 
advance’). Use of the accruals convention makes the account an income 
and expenditure rather than a receipts and payments account, although there 
are other items such as depreciation which this organization does not have 
(because it has no fixed assets) which can and normally would be included 
in accruals accounts. The use of the accruals convention is regarded by 
accountants as providing a ‘truer’ financial representation of financial events. 
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EXAMPLE 12.1 HACKNEY MINORITY ETHNIC ASSOCIATION 


HACKNEY MINORITY ETHNIC ASSOCIATION 


Income and Expenditure Account and Budget Comparison for April 


One One Favourable/ 
month's month's (unfavourable) 
Actual Budget variance 
bs = £ 

Income 
Grant: LB of Hackney - one off 4,000 4,000 _ 
1 month 2,167 2,167 = 

6,167 6,167 nil 
Income from classes 60 83 (23) 
Income from donations 100 — 100 
Income from fundraising 220 _- 220 
Total Income 6,547 6,250 297 
Less: Expenditure 
Furniture & equipment 3,700 4,000 300 
Salaries and N.1. 1,470 1,472 2 
Staff recruitment 650 75 (575) 
Rent, rates, light & heat 200 200 _— 
Telephone & postage 6 50 44 
Print, stat & publicity 90 78 (12) 
Insurance 17 17 - 
Audit fee - 33 33 
Travel Expenses 15 53 38 
Training & conferences 40 42 2 
Mother tongue teaching 100 166 66 
Fund raising expenses 20 _— (20) 
Bank charges _ 7 KY 
Sundry expenses 14 57 43 
Total expenditure 6,322 6,250 (72) 
Excess of Income over 
Expenditure (surplus) nil 225 


HACKNEY MINORITY ETHNIC ASSOCIATION 


Balance Sheet at 30th April 
2 3 

Fixed Assets nil 
Current Assets 
Prepaid rent, rates etc. 400 
Prepaid insurance 183 
Cash at bank 4,025 
Cash in hand (petty cash) 50 

4,658 
Less: Current Liabilities 
LBH grant received in advance 4,333 
Mother tongue teaching 100 

(4,433) 
NET ASSETS (“Wealth” of the group) 


225 


This is made up of: 
Accumulated fund on April 1st. nil 
Add: Excess of Income over 

Expenditure (Surplus) for the month 


Accumulated fund on April 30th 


(Source: Fairbairn and Holness, 1992) 
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Secondly, we can note the use of management accounting in the breaking 
down of the annual budget of the organization into a monthly income and 
expenditure budget which is then compared to the actual figures for the 
period. The difference between the expected and actual figure is the vari- 
ance which is used to identify areas where expectations are not being ful- 
filled and which may require management action. In this case, income has 
exceeded expectations by £297 and expenditure has been more than antici- 
pated by £72, giving a favourable variance of £225 which has been carried 
to the balance sheet. However, the largest negative variance comes from 
the expenditure on staff recruitment costs, which would have caused total 
expenditure to be more than expected by a much greater amount, were it 
not for the lower than expected expenditure on equipment (a type of capi- 
tal asset, though accounted for here as though it were a running cost). 


Thirdly, we note from the balance sheet that the bulk of the organization’s 
financial assets are held as ‘liquid’ (that is, instantly available) deposits at 
the bank. 


There are a number of other things we can note from the accounts depend- 
ing on which question we are trying to answer. For example, did the or- 
ganization or the grant giving donor (the London Borough of Hackney) 
expect there to be staff recruitment costs and was allowance made for this 
in the size of the grant? If not, how is the shortfall to be made up? Is it 
acceptable to purchase capital (fixed) assets from what appears to be a 
revenue (i.e. running costs) grant? If not, the issue of how to finance a 
shortfall will again arise. 


12.3.2 A LARGER NGO 

Example 12.2 concerns Environment 2000, a medium-sized NGO in Zim- 
babwe concerned with conservation. It was started in 1990 when the two 
founders rode bicycles from the UK to Zimbabwe to raise awareness and 
money to conserve rhinos. It now runs a number of projects related to the 
environment (Clean Up Zimbabwe, Rhino Fund, Trees For Life, Educa- 
tion and Recycling Programmes etc.). The extracts presented here are not 
the full published accounts but consist of the income and expenditure ac- 
count and balance sheet, the schedule of expenditure and some summar- 
ized information about project income and expenditure. Other information 
is provided in the accounts and annual report which accompany them (but 
are not reproduced here) which enable me to answer some of the questions 
I feel should be raised on the basis of the selected information for example, 
the notes referred to in the balance sheet. These accounts would be more 
useful for informing public action than for organizational management. 
Thus, for example, they might be useful for an investigation by a potential 
donor into the range of projects the organization supported and how finan- 
cially secure it was. 


So what does a brief analysis of these accounts tell us? Well, looking first 
at income, we can see that the bulk of it comes from donations and sub- 
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EXAMPLE 12.2 ENVIRONMENT 2000 (ZIMBABWE)(EXTRACTS) 


ENVIRONMENT 2000 


INCOME & EXPENDITURE ACCOUNT 
YEAR ENDED 31 MARCH 1996 


INCOME 1996 1995 
zs zs 
Management fees - RAP & Other 159439 0 
Donations and subscriptions 238416 264582 
Interest 5244 447 
ELP royalties less applicable expenditure 56750 68865 
459849 333894 
OTHER INCOME 
Sales and promotions 4220 2561 
Deduct: Cost of sales 
Opening stock - Rhino T Shirts 6084 7206 
Less stock 31 March 1996 3564 6084 
2520 1122 
Gross Profit 1700 1439 
Total Gross Income 461549 335333 
Deduct expenses as [below] 448245 263854 
Excess of income over expenditure for the year 13304 71479 
Fund balance at 31 March 1995 121640 50161 
Add: projects income transferred to fund 7876 
142820 121640 
Add: Bedford Truck transferred out of Education 
Fund at donated value as at 31/3/95 150000 
Less: 
Accelerated amortisation of Bedford Truck to 
estimated residual value (47500) 0 
Fund at 31 March 1996 "245320 ——=«121640 
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ENVIRONMENT 2000 


BAI HEET 
31 MARCH 1996 
NOTE 1996 1995 
zs zs 
Environment Fund 245320 121640 
Represented by: 
Fixed assets 3 97256 182509 
Trade Mark 4 300 300 
97556 182809 
Current assets 
Stock 3564 6084 
Cash accounts 76533 92406 
Debtors and deposits 117451 48372 
197549 146862 
Current liabilities 
Creditors 5 36659 42771 
Project Creditors 13126 165260 
49785 208031 
Net current assets 147764 (-61169) 
245320 121640 
CHAIRMAN 
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ENVIRONMENT 2000 


SCHEDULE OF EXPENDITURE 
31 MARCH 1996 


Administration 

Advertising and promotions 
Bank Charges 
Depreciation 

Entertainment 

Housing allowance 

Light and water 

Motor vehicle expenses 
Printing and stationery 
Rent and house maintenance costs 
Sundry expenses 
Telephone and postage 
Commissions paid 

Petty Cash 


Add: 

PROJECTS EXCESS EXPENDITURE OVER INCOME 
Painted Hunting Dog 

Tree Fund 

CleanUp Zimbabwe 

Rhino Day and Rhino Fund 

Clean Up Zimbabwe - Direct Mail 

Fun Run 

You & Your Environt. Competition 96/97 Exp in Debtors 
Chingele Clinic 

Guide Books 

Communications and Public Relations 

Eco Tourism 

Recycling & Anti Litter Programme 

Education 


TOTAL 


(Source: Environment 2000, 1996) 


1996 
zs 


186090 
1411 
2634 

52753 
5068 
7584 
3130 

25995 

19416 

60641 

20768 

(4942) 
ty) 
(403) 
380145 


1995 
zs 


135572 
756 
1632 
35331 
4254 
3800 
5986 
15965 
6164 

(1749) 
9448 
14431 


231802 
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scriptions (Z$238 416). But management fees for the RAP (Recycling and 
Anti-Litter Programme) are also significant at Z$159 439. In fact, this in- 
come is largely a grant from the Dutch Government for running the pro- 
gramme (hence the absence of a comparable figure for 1995). Then there is 
interest, which at Z$5244, is 6.2% of half the difference between the two 
cash account balances for 1995 and 1996 shown on the balance sheet (i.e. 
very roughly the presumed average balance during the year). This is crude 
but provides a plausible check that funds have not been substantially run 
down during the year and run up again in time for the end of year accounts. 
Finally, there are ELP (Environmental Labelling Programme) royalties less 
applicable expenditure. The royalties arise from businesses paying to have 
the logo on their products. This scheme has suffered a drop in income of 
17% in 1996 compared to 1995. A small amount of income arises from 
sales of T-shirts and recyclable materials. 


Expenses, under the ‘Schedule of expenditure’, show an increase in total 
of Z$184 391 over 1995, The main components of this seem to be adminis- 
tration (+Z$50 518), rent and maintenance (+Z$62 390), increased deficits 
on projects (+Z$36 048) and increased depreciation (+Z$17 422). The lat- 
ter is shown in the notes to the accounts to arise from the increased value of 
fixed assets (some of which had been donated in the year). Other 
expenditure items which might be queried include the bracketed items 
relating to the house, indicating some reallocation of expenses between 
periods and the adjustments to the projects to give a number of them net nil 
balances (especially the recycling programme). 


Returning to the Income and Expenditure Account, note that the acceler- 
ated amortisation of the Bedford truck (reducing the expected useful life of 
the truck and hence its current value) is shown as a separate expense item 
on the income and expenditure account (Z$47 500). It is rather unusual to 
take the net book value of an asset like a truck through the income and 
expenditure account in this way. The truck has not been sold, so treating it 
as ‘income’ when all that has happened is that it has been transferred from 
the education fund might be considered somewhat misleading. A more con- 
ventional approach might be to deal with it through a fixed asset account. 


But this account is not solely an income and expenditure account, since the 
surplus of income over expenditure for the year (Z$13 304) is added to 
the accumulated fund, together with some project income transferred from 
elsewhere (for which there is no explanation) and the net value of the do- 
nated Bedford truck to arrive at the total for the fund of Z$245 320. This 
figure appears again at the top of the balance sheet. It can be seen that the 
bulk of the increase in the size of the fund over 1995 is represented by the 
truck — only Z$21 180 arises from the surplus on ordinary activities plus 
the project income, some 4.7% of its expenses. This may not matter, since 
the purpose of the organization is to improve the environment not increase 
its assets. But the accounts enable us to gain a broad understanding of the 
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activities of the organization, how vulnerable it is to fluctuations in funding 
and how good its financial administration is. A small surplus of this kind, 
subject to clarification of some of the items (and subject to the audit — see 
Section 12.4 below), seems to indicate a certain management capability to 
run the organization satisfactorily. 


The Balance Sheet enables us to check this judgement. We note the signifi- 
cance of the fixed assets (mostly the truck) and the Z$76 533 in cash ac- 
counts. But one major uncertainty is the status of the ‘debtors and deposits’ 
item of Z$117 451. These are not broken down separately into money actu- 
ally received and deposited against money owed. If any sizeable proportion 
of the figure is owed, we would need to find out who owes it and when it is 
likely to be p aid. A common audit technique is to contact debtors and ask 
them to confirm the amount they think they owe. The figure reported is 
over double that of the previous year (+Z$69 079) and the reported surplus 
is only just over Z$21000 — this may be worrying unless a good expla- 
nation is forthcoming. Creditors at just over 11% of annual expenses seems 
a plausible figure (almost 6 weeks’ worth of transactions). Stock (of T-shirts) 
has run down, but is insignificant. Compared to last year, there has been a 
significant increase in net current assets. It would appear that additional 
funds have been forthcoming (but may not yet have been paid) to clear last 
year’s project creditors. Possibly it is part of the Dutch Government grant, 
which we can expect will be forthcoming. But it would be our main concern 
with this organization, based on a quick examination of its accounts, to 
understand the composition of the debtors’ figure. Given its importance, it 
is perhaps surprising that no further information about it is given in the 
accounts. 


12.3.3 AN INTERNATIONAL NGO 

This final NGO example concerns Oxfam, a major international NGO. Of- 
ten the more complex an organization is, the more complex its accounts are 
and the less amenable to a quick examination. But this does not mean that 
pertinent questions cannot be raised. In this case, we are using only a sum- 
marized version of the accounts from Oxfam’s Annual Report. This is not 
entirely fair, since the full accounts would answer some of the questions 
which can be raised. But this summary has clearly been prepared by Oxfam 
to provide an overview of certain aspects of the organization’s activities 
which are expected to be of some interest to a general reader. 


Thus what Oxfam chooses to put in a published summary (available on the 
Internet) tells us something about the kinds of question it expects readers to 
be interested in (and perhaps something about the kinds of question it hopes 
people won’t ask). Thus these accounts are primarily useful to inform pub- 
lic action. We might want to know about how extensive Oxfam’s projects 
are, how reliant these projects are on donor funding, what kind of projects 
are being supported and so on. The explanatory introductory paragraph gives 
some additional information, and I will refer to this in my discussion. 
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EXAMPLE 12.3 OXFAM 


Oxfam’s Annual Review 1995-1996 (extract) 
General finance 
Income holds up in a difficult year 


Oxfam’s general income for 1995-6 remained stable at £58 million, 
despite poor fundraising conditions. We are grateful to our volun- 
teers and supporters for their tremendous efforts. Restricted income 
(from emergency appeals and other sources) was down on the previ- 
ous year, however, so total income was £89.2 million: 11 per cent 
down on last year. In order to sustain our overseas programme at its 
current level, £4.6 million was drawn from our limited reserves. Our 
direct charitable expenditure was £82.4 million. 


[there follows a paragraph of acknowledgements] 


Where the money came from 


Donations £33.6m 
Shops & Trading £15.7m 

UK Government £13.1m 
European Union £8.9m 

UN agencies £7.56m 
Oxfam International members £2.7m 
Other agencies £1.04m 
Gifts in kind £6m 
Miscellaneous income £0.6m 
Total income £89.2m 
How the money was spent 

International programme 81% (76.1m)* 
Information/education/campaigns 7% (£6.3m) 
Fund-raising costs 9% (£8.6m) 
Administration & irrecoverable VAT 3% (£2.4m) 


* of the money spent on Oxfam's international programme, 55% was 
spent on development work, and 45% on emergencies 


Where the money was spent 


Africa 48% 
Asia/Middle East 22% 
Eastern Europe 15% 
Latin America and Caribbean 13% 
World general 2% 
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Long-term work funded by Oxfam 

Water 5% 
Health and Nutrition 10% 
Institutional Development 15% 
Information/lobbying/legal aid 10% 
Education and income generation 23% 
Agriculture 15% 
Social Organization 17% 
Other 5% 
Emergency work funded by Oxfam 

Water 13% 
Health and nutrition 18% 
Shelter 17% 
Transport of equipment 13% 
Education and income generation 9% 
Agriculture 6% 
Social organisation 4% 
Other 2% 
(from the Internet URL (address) http://www.oneworld.org/oxfam) 


This summary provides a lot of information about concerns and interests 
that readers are anticipated to have. But as a summary statement of ac- 
counts it leaves something to be desired. 


Firstly, we have no prior year’s comparison. The introductory paragraph 
says only that total income is down 11% on last year. It refers also to gen- 
eral income (which can be used for any purpose) remaining stable at £58m 
and restricted income (which can only be used for specific purposes such 
as particular emergency programmes) which ‘was down’. In order to main- 
tain the overseas programme at its current level, £4.6m was drawn from 
the ‘limited reserves’. 


But when we look at the statement of ‘where the money came from’ we 
cannot tell which of this is “general’ and which is ‘restricted’. We don’t 
know which categories have contributed most to the 11% reduction over 
the previous year. We can only guess that some of the reduction in re- 
stricted income shows up in reduced donations, but other sources may be 
affected also. 


When we look at expenditure, we can see that total expenditure was £93.4m 
and that 81% of it was spent on the international programme. We are given 
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the further information that 55% of this was on development work and 
45% on emergencies. We can’t tell how this differs from previous years — 
maybe there were fewer emergencies in the year, and that’s why the emer- 
gency income was down. Or maybe the reserves were used precisely be- 
cause there was an emergency. The main uncertainty is: how big are the 
reserves and how long could programmes be supported by using them at 
the rate of over £4m per year? We are given no information on this, nor 
indeed on any of the organization’s assets or liabilities. So whilst we might 
agree, from our earlier discussion, that it is not part of a not-for-profit or- 
ganization’s purpose to increase its wealth, the balance sheet is essential 
for such questions to be answered. 


It is clear there is a certain sensitivity to administrative costs, since the 
introductory paragraph comments that £82.4m is “direct charitable expendi- 
ture’ (i.e. excluding fund-raising and administrative costs). The separation 
of fund-raising from administration is interesting: one might presume some 
or even most of the £8.6m is paid to external fund-raisers, but it seems 
plausible that some employees also participate in fund-raising and without 
this classification, they would count as ‘administration’. And who can say 
from this disclosure how much of the funds were raised by fund-raising? 
At best, assuming all income was affected, fund-raising costs are below 
10% of all income, but if shops, overseas members and some donations 
and gifts in kind are excluded (because they didn’t rely on fund-raising 
directly), the figure rises to 15% or higher. Is this reasonable? Are admin- 
istrative expenses of 3% in line with comparable organizations? Unfortu- 
nately, accounts rarely provide this kind of information, although Annual 
Reports sometimes do. 


The next table, ‘Where the money was spent’, is of interest, but again it is 
not possible from the way the figures are presented to discover where emer- 
gency aid has been needed. The final two tables are more helpful since we 
know the split between development and emergency expenditure. Thus it 
is possible to work out how much has actually been spent in each of the 
sectors. 


So as a brief overview of Oxfam’s finances, this summary has some useful 
information which perhaps could be improved by consistent use of the 
development versus emergency categories in each table. But it tells us less 
about the overall financial health of the organization. Its activities clearly 
depend on income being maintained, but its capacity to withstand a sus- 
tained drop in income is unknown. What assets and liabilities does it have? 
Itis also difficult to assess whether its administrative and fund-raising costs 
are reasonable. The full Annual Report will provide some of this informa- 
tion, but this example highlights some dangers of relying on summarized 
information which is not the full accounts. 
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12.3.4 GOVERNMENT ACCOUNTS 

Government accounts are likely to be of interest to a researcher for differ- 
ent reasons than NGO accounts. Unless you are primarily an economist or 
banker, the national government accounts that are published are unlikely 
in themselves to be all that helpful in informing development management 
or public action investigations. Economic aggregates like GDP or fixed 
capital formation or even international indebtedness are not of primary 
interest, although these statistics will have an important impact on public 
action. What these national accounts can provide is some macro-economic 
context to public expenditure levels, how public expenditure has been al- 
located to various programmes and the extent to which these programmes 
are dependent on foreign aid or government borrowing. For example, in 
recent years there has been a trend towards project funding by interna- 
tional donors and aid agencies which can have a marked impact on a devel- 
oping economy by encouraging a reliance on emergency aid and project 
expenditure whilst on-going public services are forced to run down (e.g. 
Wuyts, 1996). 


The restructuring of the state which has accompanied new programmes of 
public management has, however, given a new significance to accounting 
as a means of demonstrating accountability and control by the new entities 
which have been created. Separation of commissioning from providing 
organizations, privatization and contracting, trading funds, income gen- 
eration and cost-recovery programmes (Gibson, 1993; Mackintosh, 1995) 
have placed more demands on departmental, programme level and fund 
accounts. Example 12.4, though from the UK, is typical of the kind of 
accounts you might expect to see in any of these new agencies and funds, 
the main difference being that, as a fund, its income and expenditure do not 
consist of many different types of item. The explanatory notes have been 
omitted, but have been used in my commentary. 


This set of accounts is for the fund which supports research into (British) 
sugar beet. It is funded jointly by the growers and the processors (British 
Sugar plc) by agreement under an Act of Parliament. The Fund commis- 
sions research from a range of research organizations, but the bulk of it is 
carried out by an experimental station which was purchased by the Fund 
but which is owned by a Trust. Certain educational activities for growers 
are also financed by the Fund; the Ministry of Agriculture, Fisheries and 
Food provides administrative support on a cost-recovery basis. 


As a fund, these accounts can give us some insight into how well this re- 
search is funded but tell us little about the organizations involved. It is 
interesting to note that the income is secured through an industry levy (on 
each tonne of sugar beet produced) on both the sugar producers and the 
processors, but that the government is a leading figure in setting this ar- 
rangement up and has supported research by providing premises as well as 
administrative support. 
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EXAMPLE 12.4 HOME GROWN SUGAR BEET (RESEARCH AND 
EDUCATION) FUND (EXTRACT) 


Home Grown Sugar Beet 
(Research and Education) Fund 
INCOME AND EXPENDITURE ACCOUNT FOR THE 
YEAR ENDED 31 MARCH 1989 
Year to Year to 
31 March 31 March 
1989 1988 
Notes £ = £ 
INCOME 
Income from statutory 
contributions 2 1,873,640 1,709,932 
1,873,640 1,709,932 
EXPENDITURE 
Net payments to research 
Establishments 3 1,291,152 1,178,309 
Net payments to British 
Sugar pic 4 326,247 337,549 
Depreciation 8 123,618 136,734 
Administrative expenses 5 64,616 45,484 
(1,805,633) 
OPERATING SURPLUS 68,007 11,856 
Interest receivable 6 182,039 107,188 
Other receipts is 93,839 125,219 
Profit on sale of fixed assets 627 995 
Amounts written off 
investments (4,823) (104) 
Amounts written off fixed 
assets (60) 
271,622 
SURPLUS FOR YEAR £339,629 £245,154 
— 
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Home Grown Sugar Beet 
(Research and Education) Fund 
BALANCE SHEET AS AT 31 MARCH 1989 
Year ended Year ended 
31 March 31 March 
1989 1988 
Notes = £ ae 
FIXED ASSETS 
Fixed assets at net book 
value 8 1,195,114 1,252,178 
Investments (cash on 
deposit) 2,604,07 2,359,036 
3,799,188 3,611,214 
CURRENT ASSETS 
Debtors 10 187,487 24,521 
Cash at bank and in hand 2,447 2,555 
189,934 
CREDITORS: amounts falling 
due within one year 11 (405,295) (394,092) 
NET CURRENT ASSETS (215,361) 
TOTAL ASSETS CURRENT LIABILITIES £3,583,827 £3,244,198 
FINANCED BY: 
General Fund as at 1 April 3,244,198 2,999,044 
Surplus for the Year 339,629 245,154 
£3,583,827 £3,244,198 
DH Andrews Ministry of Agriculture, Fisheries and Food 
Accounting Officer 13 October 1989 


(Source: Rutherford et al., 1992) 
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The research grants paid are net of certain capital expenditure on the ex- 
perimental station which is taken to the fixed asset account. Payments to 
British Sugar plc are for educational activities. Depreciation on vehicles, 
farm and laboratory equipment is written off over 4, 5 and 10 years respec- 
tively. Administrative costs include conference and travel expenses as well 
as administration and management fees for the Ministry and the National 
Investments and Loans Office which manage the financial assets of the 
Fund. These assets are substantial at over £2.6m (balance sheet), generat- 
ing £182 039 interest (around 7%). Given the expenditure on research of 
£1291 152, the reserves are enough for two years’ research (excluding ad- 
ministration etc.) and seem to be rather high. Otherwise, the levy seems 
adequate to support the current level of research, with only a small opera- 
tional surplus. 


The fixed assets include £852 052 for the experimental station because the 
Fund has a reversionary interest if it were to be sold. Debtors are outstand- 
ing payments of the levy and creditors are mostly the unpaid expenses of 
British Sugar plc in providing demonstrations etc. 


Overall, then, it appears that this Fund is operating satisfactorily, although 
the reserves seem rather high. For other accounts of this type, whether a 
fund, an income generation project or an agency, similar considerations 
apply. Where does the income come from and how is it spent? What costs 
are recoverable? How stable is the cash flow (are there large debtors or 
creditors)? Are there any stocks? What fixed assets and long term liabili- 
ties are there? What agreements or contracts are in force? Are there ad- 
equate reserves and how are they held? Who prepared the accounts and 
who audited them? Have they been appropriately approved (e.g. by the 
Board, Trustees, Minister, Management Committee etc.)? Are they pre- 
pared in accordance with accepted accounting standards? And so on. 


12.4 AUDIT 


The above examples illustrate a wide range of accounts and a number of 
questions or ways of analysing them. But so far we have not considered the 
means by which we can be confident that what appears in accounts is reli- 
able and valid. The process by which this is done is the audit, which has, 
along with accounting, become increasingly significant in institutions in 
recent years to such an extent that it is now possible to propose that we 
have become an ‘audit society’ (Power, 1997). As well as financial audits, 
we hear now about environmental audits, value for money audits, quality 
audits, medical audits, intellectual capital audits and so on. Some of the 
issues surrounding audit are similar to those surrounding accounting and 
will be briefly considered in Sections 12.5 and 12.6 below. 


A financial audit is an examination by an auditor of the evidence from 
which the final revenue accounts and balance sheet of an organization have 
been prepared, in order to ascertain that they present a true and fair view of 
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the summarized transactions for the period under review and of the finan- 
cial state of the organization at the end date. This enables the auditor to 
prepare a report to shareholders. The requirement and responsibility to 
conduct audits is specified in company legislation (for example, the UK 
Companies Act 1948 as amended) and relevant GAAP, international or 
national accounting and auditing standards. Stock markets or lending insti- 
tutions as well as shareholders will require audited accounts. Legislation 
and professional codes require that only certain firms or individuals with 
requisite expertise can carry out audits. Audit reports tend to be disap- 
pointing for anyone expecting some information about the organization 
under audit. Their wording is usually standardized and its role is confirma- 
tory —to confirm the audit was carried out in accordance with the required 
standards and that the financial statements give a true and fair view of the 
state of affairs of the organization at the balance sheet date. This wording 
applies also to NGOs. Have a look at Example 12.5. 


EXAMPLE 12.5 AUDIT REPORT FOR ENVIRONMENT 20) 


REPORT OF THE INDEPENDENT AUDITORS TO THE 
MEMBERS OF 


ENVIRONMENT 2000 


We have audited the financial statements set out on pages 
1-8. These financial statements are the responsibility of the Organisa- 
tion's management. Our responsibility is to express an opinion on 
these financial statements based on our audit. 


We conducted our audit in accordance with international Standards on 
Auditing. Those standards require that we plan and perform the audit 
to obtain reasonable assurance about whether the financial statements 
are free of material misstatement. An audit includes examining, on a 
test basis, evidence supporting the amounts and disclosure in the fi- 
nancial statements. An audit also includes assessing the accounting 
principles used and significant estimates made by management, as 
well as evaluating the overall financial statement presentation. 


Environment 2000 receives income from donations. Subject to all such 
income having been accounted for, in our opinion the financial state- 
ments give a true and fair view of the state of affairs of the Organisa- 
tion at 31 March 1996 and of the results of the year then ended in 
accordance with international Accounting Standards as adopted for 
use in Zimbabwe. 


ERNST & YOUNG 

CHARTERED ACCOUNTANTS (ZIMBABWE) 
HARARE 

23rd July 1996 


(Source: Environment 2000, 1996) 
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The first thing to notice is the reminder that the auditors are ‘independent’. 
That means, in theory, that they have no stake in the organization and are 
able to bring an objective approach to their audit. I say ‘in theory’ because 
in practice auditors rely on the management of organizations quite a lot in 
order to be able to carry out their work, and there is also the fact that they 
get paid fees by the organization for doing this. In the public sector, the fee 
issue may be obscured by audit fees being recharged by the government 
audit department or set by a regulatory body such as the Audit Commis- 
sion in the UK. But the concept of audit independence is one which can be 
problematic (Power, 1997, pp. 131-4). 


The first two paragraphs of Example 12.5 describe clearly the role and 
responsibilities of the auditor. Note, however, the importance of judge- 
ment: the auditors issue an ‘opinion’, they must obtain ‘reasonable’ assur- 
ance, they must assess ‘material’ misstatement, they examine on a ‘test’ 
basis, they ‘assess’ the accounting principles used and ‘significant’ esti- 
mates and so on. So the validity of these judgements ultimately rests on 
whether a court or another auditor considers that the audit was conducted 
in accordance with recognized standards. That is not the same thing as 
saying that the accounts are ‘correct’. Several recent large audit scandals 
such as BCCI (the Bank of Credit and Commerce International) revolve 
around such questions. The judgement delivered by the auditor is usually 
the opinion that the accounts are true and fair. Lay observers puzzle over 
why truth should require the qualification of ‘fairness’. What would true 
but ‘unfair’ accounts look like? But assessing fairness is another important 
auditor judgement. 


In the case of Example 12.5, we have what in the commercial world would 
be an unacceptable qualification: the accounts are true and fair ‘subject to 
all donation income having been accounted for’ (paragraph 3). Many NGOs 
rely a lot on donations, and if you think of the problems of ensuring, for 
example, that all street collections find their way to the right place, you can 
understand why the auditors are nervous about relying on the control sys- 
tems that may be in force that they have only examined on a ‘test basis’. 
For this to result in a material misstatement of the accounts, there would 
have to be a sizeable amount missing, but many audit qualifications are 
given on the basis of ‘uncertainty’ — the auditors are not saying that mate- 
rial amounts of money are missing, but that they were unable to satisfy 
themselves that it couldn’t happen. Sometimes written assurances will be 
sought from the management or Board and incorporated into the Audit 
Report, along the lines of ‘we have received assurances from the Board 
that the controls function as intended and subject to these assurances the 
financial statements are true and fair’. This kind of arrangement will often 
be used if there is doubt about transactions with third parties: verbal agree- 
ments, for example. So an Audit Report is a valuable safeguard but it only 
protects against certain things which have largely been defined by the ac- 
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counting profession itself as being reasonable. Auditing is a practice un- 
derpinned by the threat of being sued if you mislead, and the number of 
cases of audit litigation is rising rapidly. 


So a major conclusion from research into auditors’ work is that audit, like 
accounting, is a matter of judgement. It can be (and hopefully is) done in 
accordance with certain agreed conventions, but the existence of these con- 
ventions and the litigation which might arise through not following them 
tends to downplay the problems of interpreting accounts to give you an- 
swers to your own questions. Audit only answers, or purports to answer, 
certain questions, like ‘are the accounts, so far as the auditor can tell (using 
‘acceptable’ means of enquiry), based on agreed accounting and auditing 
conventions?’. It may not answer the questions you might be interested in, 
like ‘are the directors good managers?’ or ‘was a particular item of ex- 
penditure a good investment or use of the money?’. We can expect auditors 
to report the fact if they find evidence of unauthorized expenditure, but in 
the private (including voluntary organizations) as against the public (statu- 
tory) sector, there is no general obligation to report publicly weaknesses in 
control and authorization procedures or comment on the value-for-money 
which has been achieved. 


In the public or governmental sector, the ‘true and fair’ audit opinion is not 
so widely used because public sector accounting, except for certain quasi- 
commercial activities such as trading funds, rarely uses the full set of ac- 
counting conventions that are said to be implied by the term ‘true and fair’. 
Mostly these relate to the treatment of fixed assets. Instead, in the UK and 
elsewhere, the alternative ‘presents fairly’ is used. 


12.5 SOME PROBLEMS WITH ACCOUNTS 

This discussion has begun to take us back to some of the issues which I 
began to raise at the beginning, such as the issues of exactly for whom 
accounts are prepared and the particular problem of adopting accounting 
conventions which are primarily intended for calculating shareholder value 
in an organization which has developmental or social rather than financial 
aims. Accounting has achieved a prominent status in organizations in all 
kinds of cultures. The reasons for this are complex, but it is difficult to 
deny the power of accounting numbers, practices and practitioners in or- 
ganizational life, including governmental and not-for-profit organizations. 


Perhaps because of its growing salience, there has also been a growth in 
criticisms of accounting, from a wide range of perspectives. This is a huge 
topic and only a number of themes these criticisms have raised are outlined 
below. But they all should be borne in mind as potential limits on what can 
be achieved through analysing accounts. (For a range of different general 
criticisms and discussions of accounting see, for example, Smith, 1992; 
Jameson, 1988; Hopwood, 1988; Tinker, 1985; Lehman, 1992; Puxty, 1993.) 
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1 Shareholder versus stakeholder perspectives 
Anglo-American accounting tradition places heavy emphasis on share- 
holder value, but the general criticism that accounting places higher 
emphasis on owners than any other group of stakeholders (such as man- 
agers, employees, suppliers, creditors, company pensioners, the com- 
munity and so on) is widely recognized. 


2 Monetary values 
Accounting represents organizational activity in monetary terms. Value 
is only measured in this way. Another well accepted argument is that 
many aspects of organizations and what they mean to people, their 
achievements, even the quality of their products, cannot be captured 
solely in monetary terms. This aspect is particularly prominent in not- 
for-profit organizations, for obvious reasons. 


3 Externalities 
Accounts are accounts of organizations’ activities as seen from the per- 
spective of the organization and its management. They do not deal with 
the impact of these activities on their environment in terms of waste 
and pollution, for example. The convention of assigning monetary val- 
ues exacerbates this but is not the prime reason, since economists and 
others can calculate the costs of dealing with these problems. The ques- 
tion of the accounting entity is relevant here, since the definition of 
where an organization and its responsibilities begin and end is central 
to the issue. 


4 Accounting for what? 
The focus on monetary shareholder value and the business transactions 
which produce it obscures other assets a company will have (such as its 
employees, its reputation, its accumulated knowledge and experience) 
which are not measured routinely by accounting procedures. These as- 
sets may be especially important in not-for-profit organizations. 


5 The accounting practitioner 

The role of accountants themselves has also come under scrutiny. As a 
professional grouping with monopoly powers to carry out audits and 
accounting functions, many of which are required by law, they are in a 
very privileged position. What kind of people are they and what influ- 
ence do they have? Are they themselves the source of many of the prob- 
lems identified above? In the public sector in particular they have moved 
a long way from the traditional financial audit and now carry out value- 
for-money, effectiveness and quality audits which involve value judge- 
ments about how the whole of the organization is run. 


6 Accounting and representational practice 
Finally, there are some important questions surrounding the role of ac- 
counting in ‘making up’ organizations rather than simply ‘reflecting’, 


286 


12 ANALYSING INSTITUTIONAL ACCOUNTS 


fully or partially, their ‘objective’ reality. In this kind of criticism, ac- 
counts play a vital role in literally ‘constituting’ the organization, mak- 
ing it visible and understandable in accounting terms. In the words of 
an influential paper, ‘in communicating reality, we construct reality’ 
(Hines, 1988). Seeing accounting as a social and institutional practice 
in this way draws attention to the role of accounting as a technology, to 
its rationales and to its fluidity and historical contingency (Miller, 1994). 
Accounting ‘facts’ have great legitimacy (see Chapter 11) and the source 
of this power needs to be understood. 


12.6 NEW ACCOUNTING PRACTICES 


These problems with accounting have not gone unnoticed or unaddressed 
by accounting practitioners or accounting academics. There are a wide 
variety of attempts to improve or redefine accounting’s and accountants’ 
roles, responsibilities and practices. In conclusion, I would just like to 
mention some of them. But we should note that, with a few exceptions, 
their responses have so far done little to challenge the fundamental as- 
sumption of legitimacy that the accounting profession, mostly backed by 
powerful stock market interests, displays. 


Attempts have been made, for example, to widen the audience of accounts 
and annual reports, and to produce more information for employees and 
others. Some commercial organizations now stress their community in- 
volvement and donate funds to NGOs and other causes. Audit methodolo- 
gies have been employed in a wider range of issues. Environmental and 
social accounting has recently become a widely debated issue (Gray, Owen 
and Maunders, 1987; Gray, 1993; Mathews, 1993). The World Bank has 
revised its national wealth index to include human capital, and is develop- 
ing sustainability measures (World Bank, 1995) and new Country Futures 
Indicators (Henderson, 1996). The potential for introducing social capital 
into governmental accounts is also being explored (Jones, 1997). Some 
researchers have attempted to intervene directly in the affairs of the ac- 
counting institutions or are critically evaluating the role of accounting aca- 
demics (Neu and Cooper, 1997). The accounting profession itself is strug- 
gling to accommodate the new kinds of intellectual capital assets, brands, 
patents and intangibles that knowledge-based industries seem to be based 
on (Stewart, 1997). 


All these initiatives, together with the growth of a demand for accountabil- 
ity that seems to have plenty of momentum in it whatever its sources (Munro 
and Mouritsen, 1996; Power, 1997) highlight the importance of account- 
ing and its power. It is thus likely that, whether or not accounting as we 
currently recognize it can be transformed, pressure for more accounting in 
governments and the not-for-profit sector will continue, whether in devel- 
oped or developing countries. The opportunities this raises for conducting 
investigations are considerable, yet we need to be aware that accounting 
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data can only provide partial answers to specific questions (see Section 2 
above) and is bound to leave some loose ends unresolved. But the ques- 
tions that it can answer make the activity of analysing institutional ac- 
counts all the more likely to be needed in the future. 
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SUE MAYER 


This chapter considers how to think about and make use of data to influ- 
ence others to take action or to make changes in public policy. This style of 
research for advocacy can appear rather straightforward, but it is often 
very messy and complex, involving serious issues of legitimacy and power 
relations. However, by thinking about the problems in advance and by criti- 
cally evaluating what you are trying to do, you can better conceptualize the 
issues, the feasibility of the research and the data sources. This will im- 
prove the quality of your research and its effectiveness in changing public 
policy and public action. 


When using data to try and change or support a particular policy it is tempt- 
ing to think that it will be possible to ‘prove’ your position. You might feel 
that the data are indisputable and will inevitably convince all the commu- 
nities involved of the ‘rightness’ of your view. Such hopes are usually mis- 
placed, however. The social and political acceptance of the relevance or 
interpretation of information and the highly disputable nature of ‘scien- 
tific’ evidence often become central issues in controversies. From studies 
of controversies involving technical or scientific advances, Dorothy Nelkin 
concludes: ‘Nor is there much evidence that technical arguments change 
anyone’s mind. No amount of data could resolve the premises underlying 
the conflict over animal experiments’ (Nelkin, 1992, p. xxi). Whilst this 
should not be taken as meaning that data will never help, it does serve to 
underline some of the problems that may arise in situations where there is 
controversy over different possible actions. Although as pointed out below 
quantitative data have a special place in western policy structure, such prob- 
lems will not be solved simply by finding or collecting more data. The 
notion that more empirical information is not the simple answer many 
people hope for will be explored further in this chapter. 


Effectiveness of data demands both a proper understanding of their limita- 
tions and measures to ensure their quality and appropriateness for use. Whilst 
data alone may not be the answer, good quality data used in the right con- 
text can help gain influence for a position and, most importantly, give it the 
necessary substance to gather support. By evaluating critically the data 
that support a particular approach, and data that support alternative ap- 
proaches, evidence can be built up and support gained for your arguments. 


The way in which the framing (or conceptualization) of an issue affects the 
outcome is a theme running throughout this chapter. An example of its 
importance in development is the conflict between ‘expert’ and ‘local’ 
knowledge, mentioned by Blackmore and Ison in Chapter 2 and taken up 
further here. In a world increasingly dominated by a western version of 
logic and rationality, and where the status of the expert in industrialized 


289 


THINKING WITH DATA 


societies is growing (Krimsky, 1984), alternative approaches to understand 
or explain the world can be so different that they are discounted because 
they fall outside the accepted framework. 


We will look first at the pitfalls associated with using data. Then we will 
explore how problems are defined since the way an issue is conceptualized 
is key to shaping public policy debates and agendas. Finally we will con- 
sider practical steps to help you get the most out of your data. 


13.1 PROBLEMS WITH USING DATA 


13.1.1 DATA CANNOT PROVE ANYTHING 

When you gather data the temptation is to think there will be one straight- 
forward conclusion from it. However, the nature of data and their limita- 
tions means that any conclusions always carry an element of uncertainty. A 
range of possible conclusions can be drawn from examining even simple 
data depending on your initial assumptions. A collection of data about which 
nations received development aid, for example, is very likely to mask 
whether such aid is acting according to its aim to alleviate or eradicate 
poverty. It is easy, therefore, for competing conclusions to emerge, in this 
case to increase or cut such aid. Even as better data emerge, uncertainty 
will continue. For example, there are few gender-based data on poverty so 
arguments about how best to use development aid to improve gender equal- 
ity will always be surrounded with competing arguments. 


The boundaries implied by a question also make a difference to which 
conclusions can be drawn. Data may show that distributing money to women 
rather than men increases the production of food, from which you might 
draw the conclusion that this is the best way to spend money to improve 
nutrition. However, other differently ‘boundaried’ research may show that 
different sectors of society feel disempowered and long established social 
structures are becoming unsettled by such interventions and by the increased 
monetarization of local society, which could suggest a different conclusion 
from that formed originally. 


And of course the very methodology you use will be open to challenge. If 
survey methods are used, did these involve use of a representative sample? 
Were the questions asked openly and how were the data analysed? 


Data themselves are not knowledge. Data need to be interpreted and put 
into context before they become useful knowledge. For example, even 
though two data sets may be related, this does not mean they are causally 
related — just because children get taller as they get older and also progres- 
sively develop language skills does not mean that getting taller improves 
language skills or vice versa! 


Astriking example of how methodological arguments can lead to disputes 
is given in Example 13.1. It shows how choice of methodology can be 
shaped by your own world-view or particular priorities. 
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EXAMPLE 13.1 WHAT’S IN A METHOD? 


A dispute arose in connection with the global warming debate be- 
tween two non-governmental organizations (NGOs), the World Re- 
sources Institute (WRI) and the Centre for Science and the Environ- 
ment (CSE). 


The WRI, based in Washington DC, produced data detailing each 
country’s emissions of carbon dioxide (CO,) and other important 
greenhouse gases (Yearley, 1996, p. 102-107). The WRI report cal- 
culated each country’s emissions and then reduced the total by the 
amount that would be absorbed by natural processes. Calculating their 
emissions this way, the ranking of countries from highest downwards 
is, USA, USSR, Brazil, China, India, European Union, Indonesia, 
Canada and Mexico. What is surprising about this ranking is the sug- 
gestion that less-developed countries are major polluters. 


These results were challenged by researchers at the CSE in India 

(Agarwal and Narain, 1992). On the face of it the calculations may 

seem straightforward, and the conclusions, although uncomfortable, 

apparently correct. But the CSE made two criticisms of the data analy- 

sis. 

1 The natural processes which absorb CO, should not be apportioned 
by country but rather according to population size. 


Doing this gives recognition to the very low greenhouse impact of 
the average Indian compared to someone from the USA (0.3 com- 
pared to 4.2 on one index). Calculating things this way leads to 
India and China no longer being amongst the top five polluters, 
but replaced by Canada and Germany. 


2 All types of emissions are treated as equal. 


Thus luxury emissions associated with driving to the shops are 
seen as no different from survival emissions of using fuel for cook- 
ing, for example. 


The policy implications arising from both these criticisms are im- 
mense. 


In the first, rather than pressure being placed on India to reduce its 
emissions urgently as the WRI analysis would favour, using the CSE 
approach, India could double its total carbon dioxide emissions with- 
out reaching ‘its limit’ (Agarwal, quoted in Pearce, 1992). Thus the 
WRI could be accused of having collected and analysed data in a way 
which was against the interests of less developed nations, even though 
their approach seemed superficially logical and unbiased. 


The second, has if anything, a more fundamental effect in that it could 
support a change in emphasis of the policy itself. The shape and 
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effects of a policy to reduce CO, emissions would be very different if 
it was designed to cut luxury emissions as a priority. 


In the first case the social adjustments may be small and incremental, 
in the latter they could lead to much more social equity in resource 
use (Jasanoff, 1993). 


Example 13.1 illustrates that decisions about what data to collect and how 
to analyse them are not simple matters. There can be serious political rami- 
fications depending on which method or approach is chosen. This is be- 
cause social, political and economic judgements are involved in decisions 
upon how an issue is to be framed and what knowledge is relevant. 


All of these factors challenge the certainty of conclusions drawn from data. 
However, even if the framework of data analysis is agreed upon, there is 
also the issue of uncertainty in the raw data. In Example 13.1, the absolute 
levels of emissions in WRI’s report were uncertain and the CSE could 
simply have challenged whether countries do emit what WRI claim. Such 
uncertainty exists whatever the nature of the data — be they qualitative data 
about people’s wishes or their identification of problems, or the most tightly 
controlled scientific data. All methods suffer from data uncertainty. 


Uncertainty in data and its interpretation has been the focus of many con- 
troversies (see Nelkin, 1992) and can be used by any side in a dispute. 
Uncertainty may be used, for example, to call for an end to nuclear power 
because the effects of radiation on human health are not fully understood 
but could be serious. On the other hand it can be argued that since there is 
no proof of harm, preventing radioactive discharges may be unnecessary 
and the benefits of nuclear power lost without reason. Very often the call is 
for more research to try and resolve the uncertainty. In such situations, 
more research often only reveals more uncertainty and the demand for yet 
more research. In this highly critical model (Collingridge and Reeve, 1986), 
policy action is delayed and delayed until consensus is achieved even though 
uncertainty continues. For example, although it is widely agreed that smok- 
ing causes cancer and that lead causes brain damage in children, there are 
alternative causation theories which have not yet been discounted. In the 
cases of smoking and cancer and lead and brain damage in children how- 
ever, consensus about the link has been achieved in some countries and 
policy measures introduced to limit the risks. In contrast to these cases, the 
uncertainty surrounding arguments about whether a person’s intelligence 
is determined by genetics or the environment, remains great enough that 
policy measures which, it might be claimed, would improve a person’s 
environment and thus their intelligence, are more difficult to achieve. 


To compound these issues of uncertainty surrounding the data and the un- 
certainty of conclusions that can be drawn from those data, there is also the 
issue of ignorance. It is a factor in disputes about data, and is often much 
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more difficult to deal with than uncertainty. At least with uncertainty we 
know that a parameter exists. Even though we are unsure of its exact value, 
we have some idea of the extremes within which it might lie. However, 
with ignorance this cannot be so. To further complicate the problem, igno- 
rance tends to be denied or discounted in policy circles — with a tendency 
to rely upon an exaggerated vision of the scope and power of scientific 
knowledge (Wynne, 1992, p. 115). This restricts debate by constraining 
the definition of what are relevant questions. Before the discovery in 1985 
of the ozone hole above Antartica, it was considered rather eccentric to ask 
questions about the potential effects of ozone depletion on human health. 
The hole itself had been developing for some time, but we were ignorant of 
it. Questions on this subject before 1985 were therefore perceived as irrel- 
evant. 


One of the most innovative uses of data can be to expose the artificially 
narrowed boundaries of many policy debates. Posing new questions or look- 
ing at data in new ways may change our whole conception of a problem 
and policy responses to it (Myers, 1995). The international environmental 
NGO Greenpeace was the first to apply a research tool — gas chromatogra- 
phy mass spectrometry (GCMS) — to the measurement of chemicals in 
waste discharges in a policy context. Previously GCMS had been used to 
look at the purity of samples or for the presence of one particular chemical. 
What was significant about the new data generated was that it exposed an 
area of ignorance which contemporary policy could not deal with. The 
regulation of industrial discharges was based on a chemical-by-chemical 
approach with licenses being given which listed the levels of a list of chemi- 
cals which could be released. The GCMS study showed that there was 
often a complex cocktail of chemicals (sometimes over 100) in many in- 
dustrial discharges and some of the chemicals could not even be identified 
(Johnston and Stringer, 1989). This new knowledge makes a chemical-by- 
chemical approach impracticable and has been used to support Greenpeace’s 
policy alternative of clean production systems to give zero discharges. 


Friends of the Earth, another environmental NGO, also used new data in a 
similar way when they conducted a survey of tree health in the United 
Kingdom to demonstrate that acid rain generated in the UK was not only 
causing problems in continental Europe but was damaging trees in the UK. 
Because the UK was an important political player in the acid rain debate in 
Europe and had been denying the seriousness of the impacts, the revelation 
that its own trees (symbols of ‘a green and pleasant land’) were being af- 
fected was important in changing the political climate (Hajer, 1995). This 
was the first time that such a tree survey had been conducted in this con- 
text, so Friends of the Earth exposed an area of significant importance of 
which the policy community had previously been ignorant. 


Thus, for methodological reasons involving underlying assumptions, the 
boundaries of what is considered relevant, and how data are collected and 
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analysed, no data, or the conclusions drawn from them, can ever be con- 
sidered to be beyond dispute (see Chapter 2). Even where a consensus 
exists about a problem, this may suddenly be upset when new data on a 
previously unconsidered dimension of an issue are produced. The exam- 
ples above illustrate that data collection for advocacy can achieve break- 
throughs by reviewing the boundaries of an issue. Clearly data by them- 
selves cannot prove or disprove any argument or hypothesis. Data are the 
key, however, since their generation, analysis and exploration can suggest 
weaknesses within arguments, opinions and practices. Thus proficiency 
with data is important to your cause. Examining the evidence against your 
opinions is just as important as having evidence in your favour. As 
Mukherjee and Wuyts point out in Chapter 11, not only does it prepare you 
for policy debates, it can help you to rethink your arguments and point you 
towards a stronger case. Box 13.1 gives a useful checklist of some of the 
limitations of data as an ‘objective’ tool. 


BOX 13.1 

Data collection and interpretation will be influenced by social, 

political, economic and cultural factors. 

* Uncertainty and ignorance limit our understanding of the world. 

* Donotclaim that your data prove something — this makes your use 
of data much easier to attack. 

* Data cannot prove anything because of the variable and contin- 
gent nature of knowledge. 

Issues surrounding data can be politically significant and limit or 

extend policy options. 


* Understanding how knowledge is being shaped in any particular 
debate is important to be effective in your use of data. 


LIMITATIONS OF DATA AS AN ‘OBJECTIVE’ TOOL 


13.1.2 ALL DATA ARE NOT EQUAL 


As Box 13.1 reminds us data are not a homogenous mass of information, 
unconnected with where and how they are collected or derived. There are 
social, political and cultural forces affecting how data and the knowledge 
derived from them are perceived. For example, in some societies it might 
be difficult for a black, female, research student to have a new theoretical 
perspective accepted — even if it did fit the data available better than any 
other theory. In contrast a white, middle-aged man with a track record in 
the subject might have the same revolutionary theory accepted much more 
readily. This section explores why social evaluation is not impartial when 
it comes to the type of data used and how expert knowledge has become 
automatically accepted as fact at the expense of other forms of knowledge. 
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‘Hard’ versus ‘soft’ data 

Quantitative and qualitative data will be viewed and valued differently by 
different groups of people. Being able to count and measure phenomena is 
increasingly seen as the ‘best’ approach since it is thought to allow attribu- 
tion of cause and effect. This approach derives from what is usually re- 
ferred to as the ‘scientific tradition’ which, by valuing counting and measur- 
ing above people’s observations, makes claims to ‘objective’ knowledge — 
that it is not shaped by people’s prejudices. This is similar to the idea that 
financial acounts provide the best information about organizations since 
they are quantitative and hence ‘hard’, ‘objective’ data (see Chapter 12). 
Of course, there is a great deal of contemporary work that challenges the 
assertion that the scientific method is impartial. The interests of the people 
doing the measuring, the questions they ask and the assumptions they make, 
have all been raised as issues that challenge the image of science as impar- 
tial. (See Webster, 1991, for a useful introduction to the sociology of sci- 
ence, and Irwin, 1995 for an analysis of the complex relationship between 
science, the public and environmental threats.) 


However, despite this well established critique of ‘the scientific tradition’, 
the dominant atmosphere in policy circles is that quantitative data are in- 
variably ‘better’ than qualitative data (see Chapter 11). Sometimes people 
talk about quantitative data being ‘hard’, as in the description of ‘hard facts’. 
Qualitative data, on the other hand, are called ‘soft’ and portrayed as not 
dealing with facts. In the search for hard data there can be a tendency to 
reductionism — to cut a subject down into smaller and smaller parts so that 
the variables can be controlled and altered and cause-effect relationships 
attributed. This approach has come from the natural sciences, and is seen 
as important in the construction of ‘good science’ which is so dominant in 
certain policy cultures, including the UK (Wynne and Mayer, 1993). How- 
ever, such approaches ignore complexity and can limit the possibility of 
alternative hypotheses being accepted. For example, there has been con- 
troversy about whether pollutants in the sea can make marine mammals 
more susceptible to disease and worsen outbreaks of infectious diseases 
(Mayer and Simmonds, 1996). Acceptance of such an idea could lead to 
the policy conclusion that efforts should be made to limit marine pollution. 
However, the pollutants are likely to be having subtle effects on the im- 
mune system of marine mammals that are difficult to measure by reductionist 
approaches. The likelihood of developing disease may also be influenced 
by other factors such as nutritional status. This makes it very difficult to 
unravel the relative importance of the different factors, as there will prob- 
ably be a long series of steps in any cause and effect chain that exists. The 
marine environment is so complex that representing it in a laboratory ac- 
curately enough to produce the kinds of proof that would be demanded 
before policy action is taken to reduce pollution, would be impossible. 
This is an example where a reductionist approach could never accurately 
model the field situation. If data from such an approach are the only data 
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that have credibility, other data may be discounted or ignored even though 
they represent the only possible source of evidence on the question. 


While bias towards a particular notion of ‘hard’ science may lead to 
laboratory-based data being regarded more highly than field observations, 
it can also lead to social sciences being seen as secondary to the natural 
sciences. For example, Weale (1992, p. 18) concluded that British govern- 
ments across a wide range of policy sectors frequently play down socio- 
logical or other behavioural evidence in favour of economic or natural 
sciences evidence. The relative importance or credence given to different 
disci-plines will vary in different countries. Nevertheless, the natural 
sciences claim truths that apply the world over (Yearley, 1996, p. 85), and 
remain dominant in explaining some of the problems of sustainable devel- 
opment such as resource utilization. Therefore the culture that defines what 
is ‘good science’ with the world-views contained in the definition, gives 
rise to a pervasive problem in ensuring that data lying outside the current 
perception of ‘good science’ are taken seriously. 


It is therefore important to be aware of how different types of data will be 
viewed. As we discussed earlier, in policy debate there may be a bias to- 
wards quantitative data but, by their very nature, they cannot give the more 
subtle information you may need on how people feel or what works in 
practice. However, such qualitative data may be viewed as much more 
unreliable and open to debate and you have to be prepared for this when 
using your data. 


Local knowledge versus ‘expert’ knowledge 
Another problem is that disputed knowledge may be ‘certified’ by ‘ex- 


perts’ — lending it greater credibility than it would otherwise have. The 
increasingly technical nature of society and its problems has meant that 
use of expert knowledge is of growing importance. One consequence of 
this is that it acts against wider (and more democratic) involvement in de- 
cision-making. The justification for the primacy of experts has been elo- 
quently expressed by Stephen Yearley: “The conviction that science speaks 
objectively and disinterestedly means that one need have no qualms about 
excluding other people from decision-making since they would, in 
any event, have arrived at the same conclusions as oneself (Yearley, 1996, 
p. 118). 


Because of this the ‘high levels of mutual confidence that small élites in 
Britain have developed towards one another’ (Weale, 1992, p. 212) is not 
seen as a problem — the same is probably true in other countries. However, 
this view is becoming increasingly difficult to defend as political and eco- 
nomic interests are more explicitly seen in science. Not only are more ex- 
perts employed by industry and other partial interests, but research agen- 
das are increasingly driven by financial rather than purely knowledge- 
generating priorities. The UK public, for example, has become so cynical 
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of certain types of expert advice, that they have more confidence in NGOs 
as sources of knowledge on the dangers associated with scientific and tech- 
nological issues (Martin and Tait, 1992). 


The dominance of experts in national and international policy debates has 
implications not only for who is allowed to be involved in decisions but 
also for the type of data or knowledge that is ‘certified’ by these experts. 
For example, the way in which Andean farmers classify potato varieties is 
complex and relevant to their farming systems, yet completely different 
from that of western taxonomy (Douwe van der Ploeg, 1993). During ef- 
forts to ‘improve’ Andean potato stock, only the ‘superior’ knowledge of 
the westernized system was used. This subsequently led to problems that 
could have been avoided, had local knowledge systems been taken into 
account. The potato classification system used by Andean farmers is just 
one example of the different knowledge systems that exist to make sense 
of the world. 


Even academic studies of different knowledge systems have tended to take: 


Western ‘rationality’ and ‘scientificity’ as the bench mark 

criteria by which other cultures’ knowledge should be evaluated. 
So-called traditional knowledge systems of indigenous peoples 
have frequently been portrayed as closed, pragmatic, utilitarian, 
value laden, indexical, context dependent, and so on, implying they 
cannot have the same authority and credibility as science because 
their localness restricts them to the social and cultural 
circumstances of their production. 


(Watson-Verran and Turnbull, 1995, pp. 115-116) 


This raises particular dilemmas when using data and deriving knowledge 
from them. Western Judaeo-Christian belief is that man has dominion over 
nature, but this is not the only world-view. For example, the belief of many 
indigenous Americans is that of mother earth and a complex interdepend- 
ence between all its inhabitants — human and non-human. Whereas useful 
knowledge in the western tradition will focus more on control and manipu- 
lation of nature, for native Americans knowledge is more useful if it re- 
veals interactions and how to reduce impact on the natural environment. 


So although local knowledge may be most relevant to the design of sus- 
tainable social policies, this may have little impact in policy-making cir- 
cles because of different presumptions and aspirations for society. This 
raises some interesting dilemmas. Should you find data that will fit in with 
the preferences of the policy-maker and possibly perpetuate the downgrad- 
ing of local knowledge, or should you rely on local knowledge and risk 
lack of policy impact? 


Box 13.2 summarizes the main issues on the cultural contexts of data, data 
types and their impact. 
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BOX 13.2 CULTURAL ASPECTS OF DATA IMPACT 


© There is a policy culture in the West that favours scientific meas- 
urement and this culture is becoming dominant across the world. 

* Data which do not fit easily into the western tradition have less 
impact. 

* There is bias towards data that demonstrate cause and effect and 
this can act against theories which take a more open view of soci- 
ety and the environment. 

© There is a tendency to rely on experts to provide policy answers. 

* Local knowledge is given little value. 


13.1.3 DATA THAT DON’T SAY WHAT YOU WANT 


While there may be considerable prejudice against certain types of data 
and knowledge, these are not the only problems that advocates can face. 
One of the biggest mistakes a campaigning organization can make is to 
announce publicly that it will be collecting data to prove this or that (set- 
ting itself up for charges that any data collected are biased towards that 
view or that the mere announcement will skew any data collected, because 
people know what is expected). Avoiding announcements about what you 
are going to prove is one important lesson. 


Sometimes data do not validate your case. If you quietly collect and ana- 
lyse data and they don’t illustrate or validate what you expected, what has 
gone wrong? The first most obvious thing is that you may have been wrong 
in your initial assumptions. Data that don’t fit should trigger a critical re- 
view of your aims and objectives, and a readiness to re-evaluate them. 
Evaluating alternative explanations and counter-evidence, can, in many 
cases point towards a stronger argument than that you developed initially. 
The example given by Mukherjee and Wuyts in Chapter 11 of alternative 
explanations of famine, illustrates this. The second more complex expla- 
nation of famine, based on access to food, strengthens, not weakens, the 
case for increasing access to food in famine areas. 


Thus, lack of support for your initial argument from the data you collected 
does not necessarily mean that your overall position is weakened, though it 
certainly points to a need for reflection and reconceptulization. You may 
have picked the wrong indicator, used the wrong methodology or be trying 
to express your argument in the wrong terms. For example, finding that the 
number of deaths of cattle from a particular disease was low and so does 
not seem to support your call for more animal health provision, may not 
mean you should change your position. Other measures of cattle health, 
such as their condition, frequency with which they have calves, or the price 
they fetch at market, may be better indicators, and strengthen your argu- 
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ment. So rethinking your definition of the problem and the indicators of it 
will be an important response. 


Another example could be your use of quantitative survey methods to try 
to examine whether women feel satisfied with their role in society. This 
may have been intended to support your position that women should have 
more control and be given support to gain control. Yet answers may show 
that women accept their current position with little evidence of dissatisfac- 
tion. But deeper qualitative questions may bring out the deeper feelings 
that women may hold and find difficult to express for a variety of reasons. 
Rather than abandoning the aim of empowering women, re-evaluation of 
the type of research methodology may be a better response. 


There is a more pervasive problem, however, associated with some of the 
issues that have already been touched upon in earlier parts of this chapter. 
This is how the problem is ‘officially’ defined and therefore the context in 
which data are interpreted. For example, in policy debates you may be 
advocating that local people should be given responsibility for managing 
the land and its conflicts. But if the policy definition of the problem is that 
there is a conflict between conservation interests, tourism and local peo- 
ple’s access to land to produce food, which needs managing, you may be- 
come drawn into producing data so you can argue in this particular debate. 
Data which you may present to show how local people are most affected 
by the conflict may come across, inside the terms of the debate, as indicat- 
ing that they need ‘managing’. The terms of the debate will not have al- 
lowed the issue of who should manage land to be discussed. Rather it is 
about how land should be managed (by those currently in control). Making 
sure your position is clear when you use data is crucial if it is not to be mis- 
or re-interpreted. 


Box 13.3 summarizes some of the issues raised here. 


BOX 13.3 WHERE DATA DO NOT SUPPORT YOUR ARGUMENTS 


© Don’tannounce in advance what your research will show or prove. 

© In the light of your data, critically evaluate your policy position. 

* Reassess whether you have used the correct methodology and ana- 
lysed the data properly. 

* Question whether you have collected the most appropriate data. 

* Ask whether your data is relevant to the policy debate. 


13.2 DEFINING THE PROBLEM 


As discussed already, the types of argument and data that will influence 
people’s views will depend upon their social, political and cultural back- 
ground. Different cultures will value things differently. Political commit- 
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ments will shape the way in which people look at the problem and the 
breadth of responses which they consider plausible. For example, if power 
lies with a group of large landowners, they are unlikely to be convinced by 
data which show better yields from small-scale rather than large-scale farms. 
Aid agencies will also have their own particular organizational cultures 
which influence how they perceive problems and whether they think they 
are relevant to them. Governments use strategies such as the creation of 
new departments and the establishment of expert committees as ways of 
dealing with difficult social problems. In that way they try to gain control 
over the definition of the problem and thereby the policy responses to it 
(Weale, 1992). Being able to define the problem and have control over its 
definition gives considerable power to shape who and what is included or 
excluded. 


As you begin to think about what data you are going to use, it is important 
to understand clearly what information is considered relevant to an issue. 
The context will have evolved over time and been shaped by the actors 
involved. For example, the task of providing water to a community may 
have been addressed from a technical analysis of what was possible, while 
social issues, such as where the supply should be sited, the differing needs 
of users, and who should be involved in the decisions, were regarded as 
marginal. This will affect how data can influence decisions. 


The definition of a problem can affect whether your real feelings and 
thoughts about an issue are going to be allowed inside the debate. If they 
are not and you do not make your challenge to the framework clear at an 
early stage, this can inadvertently draw you into arguing and using data on 
someone else’s terms, which can eventually be damaging in surprising 
ways. 


For example, in the battle over the Brent Spar oil platform in the North 
Sea, Greenpeace was opposed to ocean dumping on principle, and believed 
that the Brent Spar would set a precedent for dumping of other unwanted 
oil installations. Even though their primary concern was not with the ac- 
tual effect of the Brent Spar itself on the ocean floor but rather with the 
danger of ocean dumping in general, Greenpeace became drawn into an 
argument about this one case because the UK Government did their risk 
assessment on an individual case-by-case basis. Greenpeace calculated the 
amount of oil on board the Brent Spar and then claimed that dumping the 
Brent Spar would be dangerous, even though this was not the long term 
basis of their objection. Later, a sampling error in Greenpeace’s calcula- 
tion was discovered that showed an overestimate in the amount of oil. The 
error allowed those in favour of dumping Brent Spar to portray Greenpeace 
as having unsound judgement. Greenpeace later accepted that it had been 
wrong to be drawn into a debate around the one-off case of Brent Spar, the 
terms of which they fundamentally disagreed with. 
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One common problem encountered when trying to challenge policy is that 
the boundaries exclude considerations about whether the policy is feasible 
in practice or whether it places unacceptable constraints on people. For 
example, when determining the safety of pesticides, expert committees in 
the UK argued that their assessments were correct as long as the instruc- 
tions were followed (Wynne, 1989, p. 36-37). But in calculating exposures, 
they failed to take into account the practical difficulties of using pesticides, 
for example when working with animals, how hot weather may make wear- 
ing protective clothing difficult and so on, even though this has a signifi- 
cant impact on the degree of risk. Because an expert committee had deter- 
mined these chemicals to be ‘safe’, the underlying assumptions about what 
was relevant to the assessment of safety were difficult to challenge. 


Thus it took a considerable amount of effort, the documentation of cases of 
chronic toxicity and the demonstration of the practical impossibility of fol- 
lowing safety instructions to get the possibility of harmful effects of pesti- 
cides on farm workers taken seriously. It was important in this debate that 
campaigners stuck to producing data about the realities of farm work and 
people’s experiences, rather than concentrating on laboratory data on tox- 
icity. Laboratory data alone may have led to somewhat stricter standards, 
but would not have addressed the real shortcomings in the safety assess- 
ment system which were that it ignored important parameters and thus 
drew the boundaries too tightly. 


I believe one of the most creative ways in which people can advance policy 
is by breaking down the boundaries of the debate and thereby making other 
policy options possible. However, the power of these boundaries is such 
that forcing them open is difficult; a great deal of work may be required to 
demonstrate why they are inappropriate. Box 13.4 highlights some of the 
issues surrounding problem definition and policy debate. 


BOX 13.4 PROBLEM DEFINITION AND POLICY DEBATE 


* It is important to understand the way in which a policy debate is 
framed, 

* Governments tend to restrict and boundary policy debates in ways 
that fit with their preferred policy options. 

* To have a clear understanding of the policy debate it is necessary 
to find out how all the actors define the problem. 

* Data that fit within the policy framework will more easily be taken 
seriously than data which open new issues. 


* Redefining the terms of debate can be one of the most productive 
ways of using new data, but takes considerable time and effort. 
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13.3 MAKING THE MOST OF DATA 


Your data will inevitably be put up for public scrutiny. Scrutiny will be 
most stringent where the data conflict with dominant views or, as discussed 
above, the data are drawn from sources attracting little credibility. They 
might be qualitative, based upon people’s personal observations, or drawn 
from a different discipline, such as sociology, that might not immediately 
be considered relevant to a problem of, say, pollution. Here we will look at 
practical ways of maximizing the impact of your data. There are three 
golden rules: 


1 Understand the positions of all the actors in the debate and how they 
interact with each other. 


2 Constantly reflect critically on your own position and the way to achieve 
your goal. 

3 Seek new ways to influence the debate and define the problem in ways 
that integrate your views and the data. 


13.3.1 DO YOU HAVE THE RIGHT DATA? 

The first thing to do is to ensure you possess, or are going to get, relevant 
data that support your case, and/or throw doubt on alternative cases. Who 
will collect such data is a question to be addressed later but the criteria 
for choice will influence the planning stage of research. What purpose are 
the data to serve? This is another important question that is part of the 
planning process. In making decisions about data collection, it is impor- 
tant to be careful to focus on where data are most needed or will have most 
effect. 


The kinds of questions you might ask are: 

© What do you hope the data will show? 

© What effect will they have on the debate? 

¢ Is there a better, easier or cheaper way to achieve the same effect? 

* Are you looking at the best parameters? 

© What are the weak points? 

* How will the data lead the debate forward? 

All of these questions are best addressed at the planning stage by the whole 
of the team working on the issue. Often, supporters and sympathizers will 
help you. What you are doing is finding out how the data you intend to use 
fit into your overall strategy. At this planning stage it is good to think 
holistically about the boundaries of the issue and data needs. Whoever 
does the actual research (whether yourself or a consultant), it is very im- 


portant to write a brief. This concentrates the mind, enhances critical re- 
flection and gives a basis for evaluating the work as it progresses. 
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13.3.2 WHO SHOULD DO THE RESEARCH? 

There will also be questions about where the data are to come from. They 
may derive from your own research — desk or original. Desk research is 
second hand, and reinterpreting data, which is often not in exactly the form 
you want can be hard (see Chapter 11). Gaining access to the data, espe- 
cially if it is not yet published, can also be difficult. However, doing origi- 
nal research can be both time-consuming and costly. Here the key question 
IS: 


* Do you need new data or can you use existing data in new ways? 


Not only may the type of knowledge (local or expert) influence how data 
are received in debates and hearings, but so will the identity of the pre- 
senter. The ‘independent expert’ is the voice that seems to carry most weight 
these days. These are people with no long-term financial involvement with 
the people for whom they do research and who therefore, presumably, carry 
credibility in their field. Critiques of objectivity in research will inevitably 
apply, but the influences will be more subtle and disguised, less easily 
open to challenges of partiality. Using an independent expert to give weight 
and credibility to data that might be seen as ‘soft’ (qualitative) local knowl- 
edge can be a useful strategy to make the maximum impact with groups of 
people who would normally be resistant to such data. 


Managing outside experts is not easy, however. Although using ‘independ- 
ent’ people who have undertaken the research to present the findings can 
be very persuasive, there can also be problems. For example, in public 
they may be less willing than they were in private to draw conclusions that 
challenge orthodoxy. One of the most important things is to ensure that 
there is a clear brief produced, agreed by both parties, and including details 
about payment, who has the publication rights and deadlines. These may 
seem like bureaucratic niceties, especially when you are working with peo- 
ple you know well, but in the long run they can help maintain good rela- 
tions and avoid the risk of argument through misunderstandings from oral 
communications. Example 13.2 is a short description of one NGO’s ap- 
proach to the use of experts. 


In deciding whether to use an expert the important questions to be asked 
here are: 

¢ AmI, or is my organization, qualified to do the research? 

* Is there an outside person who could do the work better? 

* How will who does the research affect its impact? 

Again these are a questions of tactics and need to be answered in the con- 


text of your overall strategy. They need to be considered by the same group 
who discussed the questions about how to collect the right data. 
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EXAMPLE 13.2 THIRD WORLD NETWORK AND THE 
BIOSAFETY PROTOCOL NEGOTIATIONS 


Background 


The Third World Network (TWN), an NGO based in Malaysia, is 
trying to influence international negotiations to establish international 
regulations governing the release, use and transport of genetically 
manipulated organisms (GMOs). These regulations would come un- 
der a Biosafety Protocol. 


The positions of important actors 


The USA and Europe attempt to steer negotiations both to avoid pub- 
lic outcry and to avoid alienating industry. Most of the less developed 
nations want strict regulations. The industries using genetic engineer- 
ing are lobbying very strongly against such regulations, on grounds 
of competitiveness and lack of harm of GMOs. The developed coun- 
tries have defined the terms of the debate as those of physical damage 
by GMOs, and technical expertise in this area is taken most seriously. 
Developing countries and NGOs such as TWN are equally concerned 
about possible social effects of the technology, such as product sub- 
stitution of developing country produce by laboratory or GMO as- 
sisted production in more developed countries. 


Third World Network’s approach 


The TWN needs to show that GMOs are potentially harmful and that 
they must be tightly regulated. The social impacts of GMOs are not 
believed by the powerful developed countries to be appropriate con- 
siderations for a Biosafety Protocol. Transnational corporations sup- 
port this approach and thus restrict the definition of the problem. 
Because there has as yet been little use of GMOs, there is no actual 
evidence of harm, so the TWN had to produce data showing that rel- 
evant experts believe there is potential for harm and thus that regula- 
tions are needed. To do this they held a conference in 1994 called 
Redefining the Life Sciences, out of which came a statement that 
underlined the potentially damaging ecological impacts of GMOs, 
especially in less-developed countries, and called for international 
regulations. Since then the TWN has gathered signatures of support 
from many scientists around the world in both developed and less- 
developed countries. These data have been used to complement other 
political work. 


Evaluating TWN’s approach 


The TWN has called upon the authority of scientists to back their 
view that GMOs are potentially harmful. Because most of these sci- 
entists have come from countries opposed to strict and holistic regu- 
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lation the hope is that it helps challenge the position of those coun- 
tries. The TWN has also established a database of critical scientists to 
comment on the technology and help TWN and other NGOs. 


The negative side of this strategy is that it shapes the issue as basi- 
cally a technical one with which scientists rather than citizens are 
best able to deal. It may also tend to add to the perception that the 
risks of GMOs are technical, not social in nature. In other words it 
leaves the framework of debate unchallenged. However, this is only 
one part of TWN’s campaign strategy. 


13.3.3 ARE THEY GOOD DATA? 


Once you have assembled your data it is important to ask whether they 
stand up to critical scrutiny. Try to be as critical as possible yourself. Get a 
sympathetic outsider who knows the area to evaluate them. Better still, ask 
a critic to give you an opinion. This helps prepare for criticism and gives 
you the opportunity to change tactics or re-evaluate your approach, It is 
time well spent. 


The questions to ask here are: 

* Have the data been collected and analysed properly? 

* Are the conclusions being drawn from them justifiable? 

* What other interpretations of the data are there? 

© What are the weak points? 

¢ Should the issue be reconceptualized? 

These questions are worth asking both at the project planning stage and 
when the project is finished. They usually demand a different group of 


people from those planning the overall strategy: technical expertise is most 
helpful here. 


13.3.4 DID IT WORK? 


There should always be an evaluation of whether a piece of work was 
effective. This is an important part of the learning process. Here the ques- 
tions to ask are: 


* What was achieved by using the data? 
How has the policy evolved? 

Did the data stand up to scrutiny? 

* How could things have been done better? 


It is often difficult to recognize when is the best time to ask such questions. 
The answer is probably that it is something to come back to repeatedly. 
Often the effect of a piece of work will not be seen for some time, or, if it is 
part of a bigger campaign, the effect of an individual part may be difficult 
to detect. You will need both the strategists and technical specialists to help 
with evaluation. 
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13.4 CONCLUSIONS 


In this chapter I have aimed to show that making the most of data for advo- 
cacy involves a range of conceptual issues and practical skills. I have em- 
phasized that data must be interpreted and displayed to reveal new knowl- 
edge and so advance your aims. Knowledge is not fixed so much as so- 
cially shaped by the questions people and groups ask and the ways they try 
to answer them. Different societies make sense of the world in different 
ways through their different knowledge systems. What is so striking at the 
moment is the way in which western scientific logic dominates debate and 
controversy over sensitive issues. Since so many issues, such as resource 
depletion, desertification and biodiversity are seen as global in nature, 
western knowledge systems are influencing decisions at all levels — 
national, regional and international. 


Although western science makes claims to impartiality and lack of bias, 
like all other forms of knowledge it is shaped by the cultures that make and 
use it. It will be influenced by the prejudices and interests of the communi- 
ties involved in its generation and use. When western science comes to 
define a problem it necessarily encapsulates western values. Because the 
definition of a problem critically influences how the issue is finally ad- 
dressed, understanding how an issue has been defined is fundamental in 
order to change it. Since great weight is placed on expertise, and experts 
tend to be chosen because they share the policy maker’s position, breaking 
open the terms of the debate to represent your own can be difficult. Data 
have to be robust — open to critical analysis — for maximum effectiveness. 


Independent experts can be enlisted to support a particular cause. There is 
increasing recognition of NGOs as experts in their own right (Weale, 1992). 
This raises other dilemmas. When using experts to counter other experts 
does this perpetuate the problem of public access and control of their en- 
vironments? NGOs are not democratically elected and simply having their 
voice heard is an inadequate substitute for adequate democracy. 


Issues of democracy may seem a long way from thinking about how to use 
data effectively. Day-to-day use of data may not, by itself, make a major 
impact on democratic processes and practices. But the use of data to influ- 
ence public policy has historically been part of a social system grounded in 
the cult of the expert, accessed by the few. Finding innovative ways to give 
voice to concerns not usually heeded by those in power is one of the excit- 
ing and enjoyable things to do with data. Data production and use have, for 
me, no merit on their own, but merely form part of a well designed cam- 
paign that allows the public to voice its concerns and understand the issues 
better, and improves the chance that all people will be able to influence 
events. 
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ALAN THOMAS 


Case studies are often used as a form of investigation aimed at informing 
development policy and public action. However the use of case studies is 
more like a framework for investigation than a specific method. Within 
that framework you can expect to use several methods, notably semi-struc- 
tured interviews but also including data analysis and surveys, if appropri- 
ate. For example, a case study of a science park could include a survey of 
the businesses in the park as well as overarching issues such as where the 
park is located and the history of how it was set up. Thus you would prob- 
ably have to combine several of the skills discussed in previous chapters in 
order to undertake a case study investigation (see Section 14.3.1). 


This chapter’s title ‘Challenging Cases’ is chosen to have a double mean- 
ing. Case studies should be challenging; in fact, the cases you choose to 
investigate should challenge your ideas as fully and broadly as possible. 
The investigation itself should be undertaken by challenging the cases be- 
ing studied, using your ideas to probe, ask difficult questions, and try to 
find satisfactory explanations. 


14.1 CASE STUDIES FOR DEVELOPMENT POLICY AND PUBLIC 
ACTION 

14.1.1 CASE STUDIES AS A GENERAL RESEARCH APPROACH 

Compared with other methods such as the analysis of generalized, second- 
ary data (Chapter 11) or structured surveys (Chapter 6), case studies are 
used in specific circumstances and to answer different types of question. A 
case study is often ‘an account and an analysis of particular events and 
decisions’ (Curry, 1992, p. 1) and can be used to ‘illuminate a decision or 
set of decisions, why they were taken, how they were implemented, and 
with what result’ (Schramm, 1971, quoted in Yin, 1994, p. 12). 


In terms of the distinction between ‘intensive’ and ‘extensive’ research 
used by Sayer (1992), and others, case studies are a form of intensive re- 
search in that ‘the primary questions concern how some causal process 
works out in a particular case or limited number of cases’ while structured 
surveys and other extensive methods are ‘mainly concerned with discover- 
ing some of the common properties and general patterns in a population as 
a whole’ (Sayer, 1992, p. 242). 


Yin (1994) compares case study method with the use of experiments and 
with historical or archive analysis as well as with survey method. He sug- 
gests three factors are important in determining which approach is appro- 
priate: 

* the form of the research question; 

¢ whether there is a contemporary event focus; and 

¢ the degree of control available to the researcher. 
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Case studies ask ‘How?’ or ‘Why?’ questions (Chapter 1) about a contem- 
porary set of events, over which the investigator has little or no control. 


By contrast, survey methods (and data analysis based on secondary data) 
tend to answer questions of the ‘What?’, ‘How many?’, or ‘How often?’ 
types. Both survey and case study methods can attempt to answer ‘Why?’ 
questions, but they do so in very different ways. Survey or data analysis is 
used to look for consistent behaviours or generalizable ‘laws’ which can 
then be applied to sets of similar examples, while the case study approach 
is used to investigate causality directly by looking in detail at how the 
causal processes work within particular cases. 


Yin’s analysis also shows up the differences and similarities between case 
study methods and the use of experiments. In an experiment the researcher 
controls certain conditions in order to see if the processes which are ob- 
served occur in accordance with preconceived theoretical ideas or not. In 
choosing a case study the researcher cannot control the conditions pre- 
cisely but can still try to find a case which will provide evidence for or 
against a particular theoretical idea. It is often more appropriate to think of 
case study research as a series of uncontrolled experiments, rather than as 
a survey with a very small sample. 


14.1.2 USES OF CASE STUDIES: CLASSIFICATION AND EXPLORING 
CAUSALITY 

In what at first may seem to be a somewhat different characterization than 
Yin’s, Langrish (1993) suggests that case studies can be used to differenti- 
ate between classes or types of phenomena. However, answering a ‘What 
types?’ question is actually closely related to answering ‘How?’ questions. 
Langrish argues ‘If case studies can be compared with biological studies, 
then the purpose of case studies becomes clear, namely to assign the differ- 
ent examples into ‘classes’ and to observe the different ways in which the 
different classes ‘survive’ in either the same or different environments’ 
(1993, p. 3). 


Langrish continues by suggesting four aims for case study research: 

1 to develop labels for a classification scheme; 

2 to look for principles underlying a taxonomy; 

3 to look for movement through time: for example, how things adapt, 
how control is exerted; 

4 to unravel causation, especially relations between purposeful behav- 
iour by several actors or agencies. 


As you probably notice, the first two aims relate to classification but the 
last to causality. It is useful to understand how closely the two kinds of aim 
relate to each other. 


Langrish terms case study methods ‘biological’ and argues that their pur- 
pose is to assign examples into “classes’ and to observe different ways in 
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which members of different ‘classes’ ‘survive’ in the same or different en- 
vironments. He suggests that cases grouped into classes are analagous to 
animals grouped into species. The way to understand how different cases 
differ is to observe in detail how they survive, in other words their internal 
workings and their relationship with their environment. Each different type 
is associated with a different answer to the ‘How?’ question and with a 
different pattern of causal relationships. This type of reasoning can be ap- 
plied to many different contexts: how different firms succeed in different 
competitive environments, how households survive by adapting different 
livelihood strategies in response to policy changes, etc. 


Research reported by Dina Abbott in Chapter 9 gave an example where 
individuals and their livelihoods were treated as case studies rather than as 
members of a group with basic similarities that could be surveyed. Abbott 
studied in detail the activities of khannawalli (women who provide meals 
for migrant workers on a regular basis). She did this in order to answer the 
question: ‘How do khannawallis make a living from this?’. In doing so she 
found a number of types of survival strategy which varied according to 
‘internal’ factors such as family labour and other resources available as 
well as ‘external’ factors such as other income opportunities in the house- 
hold, social relations (including links to rural kin), and indebtedness. Ex- 
ample 14.1 shows in another context (a study of worker co-operatives in 
the UK) how using case studies to answer a ‘How?’ question can lead to 
the generation of a typology or classification. 


EXAMPLE 14.1 ‘DEVELOPING SUCCESSFUL WORKER 
CO-OPERATIVES’ IN THE UK 


During the late 1980s I was part of a research team working on a 
project which started by asking how well worker co-operatives per- 
form in the hope of suggesting ways of promoting more successful 
co-operatives (Cornforth et al., 1988). There were several meanings 
of ‘success’ in effective use by different co-operators, ranging from 
the promotion of democratization in society and other radical social 
goals through simple survival to conventional measures of business 
performance. 


The study combined a comparative data analysis element, in which 
we measured business and survival indicators for co-operatives in a 
number of sectors, with a number of case studies, which we then used 
to explore how co-operatives survived, why some failed, and how 
those that succeeded in various ways managed to do so. 


We took a pragmatic view of what constituted success, including sur- 
vival over a certain reasonable period, which required both a mini- 
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mum business performance or better and at least some maintenance 
of other goals. We found a variety of strategies corresponding to 
different types of co-operatives, and for each type we were able 
to suggest likely future pathways including different ways in which 
each type might fail as well as continue to succeed. 


The ‘preliminary typology’ used to obtain a spread of types among 
the case studies was based on the origins and dominant motivation 
behind the formation of the co-operatives to be studied. Thus a number 
each of ‘rescue’, ‘endowed’, ‘philanthropic new-start’, ‘alternative’ 
and ‘job-creation’ co-operatives were included. However, looking at 
how co-operatives achieved success and their strategies for survival 
led us to concentrate on how they maintained commitment, both from 
their financial backers and from the members themselves. By the end 
of the study we were able to reconceptualize the meaning of success, 
and define a new typology based mainly on economic performance 
and ‘orientation’ in which each type maintained commitment in a 
different way. The four types we differentiated were: 


1 Marginals: economically marginal, job and socially oriented co- 
operatives, consisting of peripheral workers with low or underval- 
ued skills, working at low wages and low productivity, but main- 
taining commitment through providing social benefits and because 
of lack of alternatives. 


2 Radical marginals: economically marginal but socially or politi- 
cally radical co-operatives, often with potentially more highly 
skilled workers choosing to pursue social goals on a non-profit 
basis, with relatively low wages but maintained by combining 
shared political commitment and moral orientation with high la- 
bour turnover. 


3 Instrumentals: more economically successful, job-oriented co-op- 
eratives, with higher-skill workers more like a cross-section of the 
general workforce, success seen in terms very similar to a conven- 
tional business, and maintenance of commitment requiring a com- 
bination of material rewards with opportunities for participation 
and non-material rewards for those taking on leadership roles. 


4 Pathfinders: high economic performers with mixed orientations, 
still evolving, with continuing economic and organizational self- 
development, combining moral with instrumental orientations and 
material with non-material rewards, and trying to maintain a con- 
tradictory position of challenge to conventional capitalist working 
and management practices. 


(Cornforth et al., 1988, pp. 202-203) 
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14.1.3 WHEN TO USE A CASE STUDY RATHER THAN A STRUCTURED SURVEY 
Case study approaches are different from — though not in any way inferior 
to or less rigorous than — other research methods. They are different not 
only in the types of questions addressed but in the contexts in which they 
are most practicable. 


It is particularly important to understand when a case study approach is to 
be preferred to a structured survey. As noted in Chapter 6, to undertake a 
survey you have to be able to define the total ‘population’ about which you 
want to find out something. By studying a sample of the members of the 
population and analysing the results, you make a statistical inference about 
the properties of the population as a whole. In order to consider case 
studies as an alternative method, it helps to generalize from the notion of 
‘members’ of a ‘population’ to think in terms of ‘units of analysis’ within a 
‘universe’ which you need to study. While ‘members of a population’ (e.g. 
firms within a sector; households in a village) are generally treated 
as constituting a group with basic similarities that could be surveyed sepa- 
rately, there could be several possible ‘units of analysis’ at different levels 
(workers, departments, networks of suppliers and subcontractors as 
well as simply ‘firms’; individuals, gender relations, livelihood systems, 
‘patron-client’ units and kinship groups as well as simply ‘households’) 
whose characteristics and interrelationships could be analysed within the 
‘universe’ of a particular sector or village. 


Consider the example in Chapter 11 about the effect of structural adjust- 
ment on firms in Tanzania. If you had to follow up the data analysis with a 
study about changes in the small scale manufacturing sector in Tanzania as 
a result of structural adjustment, would it be better to do so with a survey 
of small manufacturing firms or a number of case studies? A structured 
survey might do the job, but only if 

* itis clear that the important unit of analysis is the firm; and 


* you can answer your research questions from aggregate and compara- 
tive information about firms as individual units; and 


* you have sufficient information about the population of firms within the 
sector; and 


* you can get access to the statistical sample you require (see Section 
14.2.1 on difficulties associated with sampling). 


If, however, you have research questions about how and why firms react in 
different ways to the new policy regime, which require looking within firms 
at how strategy is formulated, then you may prefer a number of more de- 
tailed case studies of firms and their inner workings. Or your questions 
may lead you to define a smaller unit of analysis than the individual firm, 
such as decisions and how they are made, and your research could focus on 
case studies of particular decisions. Alternatively, again depending on the 
particular research questions being asked, you might want a larger unit of 
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analysis, perhaps case studies of groups of firms in particular localities or 
sectors which would analyse how they interrelate as well as how the firms 
within the group operate. 

14.1.4 ‘CHALLENGING CASES’ IN POLICY RESEARCH 

So far we have been considering case studies as a general research ap- 
proach alongside others such as structured survey. However, what about 
the use of case studies as an investigative method for development policy 
and public action? Are there ways in which case study method — and the 
notion of ‘challenging cases’ — is particularly appropriate for policy-ori- 
ented and action-oriented research? 


Note that in a sense a particular policy or policy process is a case. Thus 
evaluating and monitoring a policy as it is implemented might be regarded 
as case study research in itself. There are many examples of the study of 
particular policies in this book, including the study of agricultural exten- 
sion in northern Mozambique in Chapter 3, science and technology policy 
in Poland in Chapter 8, and the work of the South African Labour Market 
Commission in Chapter 10. These types of case studies are not exactly 
what I am looking at in this chapter. I am not concerned with the investiga- 
tion or ongoing monitoring of a whole policy area, but with various sce- 
narios where a development manager will choose to investigate a specific 
case study in order to inform policy choices. The case to be investigated 
could be much smaller in scope than the whole policy area; alternatively, it 
could be a similar case elsewhere (geographically or institutionally outside 
the development manager’s own concerns) but nevertheless with potential 
to contribute something from which to learn (see below for examples of 
different relationships of case studies to the overall policy concern). 


In keeping with the general approach of the book, the type of scenario I 
have in mind is where a development manager is considering a course of 
action and has a set of ideas which appears to justify (to himself or herself 
at least) that course of action. ‘I think we should do so-and-so because X 
will lead to Y and the effect or result will be ... ’, and so on. However, if 
challenged (that word again) there may not be sufficient or strong enough 
evidence to support (or disprove) the development manager’s ideas. So 
more, or better, evidence, is required and the development manager must 
be open to evidence which suggests alternative ideas. If the evidence re- 
quired is about the kind of questions which case studies are good at ad- 
dressing (‘How?’ and ‘Why?’ questions particularly) and the type of phe- 
nomena which need investigating are such that the generation and analysis 
of survey data is inappropriate (as with events, processes or decisions), 
then the need may be to investigate a ‘challenging case’ or several ‘chal- 
lenging cases’. 


This is not the only scenario. The investigator may well be distinct from 
the manager or the agency behind a particular policy — either commis- 
sioned to be independent or undertaking research from a separate, possibly 
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critical, institutional position. Still, whether you are the investigator your- 
self or are commissioning research from others, you should develop your 
ideas before choosing a case or cases to study or writing the brief for a case 
study investigation. (How else can the case be challenging? As Yin (1994) 
says, case study design should be theory-led.) Your ideas will still be ideas 
about what to expect from the policy situation or process being investi- 
gated, and the above point stands about the need to conduct a case study 
investigation to get evidence that would challenge (or validate) the ideas. 


Cases to be studied may have different positions in relation to develop- 
ment management and the policy process. Possibilities include: 


1 cases of the ‘object(s)’ of policy interventions (cases of particular indi- 
viduals, particular communities, villages, regions, firms, industries). 
Abbott's study of khannawallis in Chapter 9 and the case studies of 
particular co-operatives in Example 14.1 are of this type. If policies are 
already in place, such case studies could look at the impact of policies 
and the processes by which the particular ‘object’ has been affected by 
those policies. If it is a question of assessing potential policy inter- 
ventions in advance of implementation, the study might concentrate 
on existing ‘coping’ strategies or on interrelationships, with a view to 
asking how these might be affected by the new policy. 


2 cases of similar policies or interventions elsewhere (in another region 
or country, by another department or ministry, involving a different set 
of non-governmental organization (NGO) partners, etc.). Although he 
did not undertake the study himself, Hanlon’s example in Chapter 3 of 
using a case study about US sanctions against Cuba to inform work on 
how to make sanctions against apartheid South Africa more effective is 
of this type. Here the focus could be on how far the policies there worked 
as it is envisaged the policy here should work, or on noting problem 
areas in order to check if they are likely to be problems here given the 
different circumstances. 


3 cases of the policy development process itself (this could be a case of 
the process ‘elsewhere’, or of a previous process involving your agency). 
Examples include the case study in Chapter 1 of the District Primary 
Education Policy (DPEP) in India. The cases in Example 14.2 are of a 
somewhat similar type, being studies of NGO influence on environ- 
mental policy. Example 14.3 gives some detail of one of these studies, 
on a campaign against a particular water development project in Bot- 
swana which forced a policy change. Such studies are likely to show 
how a multiplicity of agencies is involved over time and could inform 
the way the process of consultation and negotiation is managed in rela- 
tion to policy development in the future. 


4 a ‘multi-level cut’ (for example, the process of policy development, 
plus how implementation occurred in a particular region or two, plus 
how it impacted on a small number of communities and individuals 
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within them). An example here, in the context of an investigation of 
how advice is given to small businesses in a country, would be a case 
study of the development of a particular policy, say towards state sup- 
port for certain types of local advice agency, plus one or more case 
studies of the workings of particular advice agencies, plus several cases 
of individual firms and their relationships with those agencies. 


Whatever the relationship between the cases to be studied and your policy 
concerns, it is important to have ideas to guide your case study investiga- 
tion. For example ‘a case study of Firm X’ doesn’t sound ‘challenging’ — it 
implies simply just finding out all you can about that firm rather than using 
the case for evidence in a focused way. It is better to conceive case studies 
from the start in terms which reflect the ideas you want to challenge. For 
example, a case study of ‘how Firm X has reacted to structural adjustment 
successfully enough to survive’. 


14.2 CHOOSING CASES WHICH CHALLENGE IDEAS 


Having decided that a case study approach is appropriate, how should you 
decide on the case or cases to study? Since the boundaries of a case are 
never clear, this is a question of ‘framing’ (i.e. which aspects should be in 
the picture) as much as of choice. 


Before looking at different strategies for using case studies, it is worth 
being very clear that it is not a question of finding a (small) representative 
sample. Sampling is a concept that belongs with structured survey method 
and requires that the ‘universe’ of investigation can be defined as a statisti- 
cal population from which a sample can be taken. Choosing case studies is 
not like this. 


14.2.1 CHOOSING CASE STUDIES IS NOT LIKE SAMPLING 

Once we start thinking about case studies as focusing on units of analysis 
within a population or universe, there are several reasons why the idea of a 
representative sample is likely to be inappropriate or impossible. 


1 There may in principle be no enumerable ‘population’ from which to 
sample. 


Studies of policy often hit this problem, particularly since it is well 
argued that the policies of many agencies depend as much on what does 
not get on to agendas as on what is actively debated and decided. How 
could one conceive of examples of ‘non-decision-making’ (Crenson, 
1971) being ‘representative’? Representative of what? To give another 
example, it would have been very relevant to the research in Example 
14.1 to study attempts to set up co-operative businesses in order to see 
what distinguished successful attempts from unsuccessful ones. But there 
was no way of knowing the extent of the ‘universe’ or enumerating the 
‘population’ of all such attempts. How far down the line would discus- 
sions have to go before being regarded as an ‘attempt’? And there is 
probably no record of most unsuccessful attempts in any case. Thus 
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although case studies of attempts could be included in the research, 
once again they could not be chosen as a representative sample. 


You may not know enough about the units of analysis in the ‘universe’ 
to be able to define a sample. 


There are often occasions when you want to focus on individuals or 
examples with particular characteristics, but available data does not tell 
you which they are. For example, Dina Abbott (Chapter 9) could not 
have defined a representative sample of khannawallis because there 
was no data on which women are khannawallis, and clearly a statistical 
survey of the membership of an organization such as the Annapurna 
Mahila Mandal (AMM) would not have given a representative sample 
of khannawallis as a whole since many khannawallis were not mem- 
bers of the AMM. This type of problem can also occur if you would like 
a stratified sample but data on the variable on which you would wish to 
stratify is not available. In effect you have to undertake the study in 
order to gain the knowledge that would enable you to define a sample. 
For example, to do a study of the effects of participative labour rela- 
tions on firms you might want a sample stratified according to whether 
or not such participative relations are present, but this obviously could 
not be defined before some research to determine what does or does not 
count as participative. Any sample representative of firms in general 
might contain very few with what the study ends up defining as 
participative labour relations. 


You may not have access to part of the population. 


If parts of the population cannot in practice be included in any sample, 
then this may effectively bias your eventual choice of cases so badly as 
to render the idea of sampling useless. Again, Chapter 9 provides ex- 
amples of this problem, such as where Abbott was unable to interview 
scheduled caste men although they would have formed an important 
part of any sample of the clients of khannawallis, and this was one 
reason for her rejection of survey method. In Example 14.1 there was a 
slightly more subtle version of this problem which can occur with any 
study aiming to uncover reasons for ‘success’. Those co-operatives which 
regarded themselves as failing would be less likely to allow access to 
the study team than ‘successful’ co-operatives, for the understandable 
human reason of not wanting their failures exposed. Thus any study 
undertaken by interviewing in a sample of co-operatives would be likely 
to be biased to an unknown extent in favour of success stories. 


You may be specifically interested in rare or unique phenomena. 


A recurring problem in policy research is how to study examples of 
“good practice’ where most examples are bad. Similarly, how can you 
investigate why certain courses of action, individuals or organizations 
are ‘successful’ when the majority are not? Yin (1994) gives the exam- 
ple of a study of the introduction of information technology into schools. 
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A survey would have shown a depressing picture of failure and might 
even have failed to include any success stories at all. If there is even one 
example of success, it is surely more useful to study it in depth than to 
dismiss it as unrepresentative. Indeed, certain topics of investigation, 
such as innovation, by their nature occur rarely. Not surprisingly, a sur- 
vey of NGOs in Zimbabwe showed very few of them engaging in inno- 
vative service delivery (Vivian, 1994). Vivian’s conclusion that innova- 
tion by NGOs is unimportant seems too strong — since innovation in a 
few key NGOs might well show a substantial impact. Similarly, study- 
ing a phenomenon which is new requires studying cases which cannot 
yet be chosen as representative of anything. 


5 There may be many independent variables that could occur in a very 

large number of possible combinations compared to relatively few ac- 
tual examples in the ‘population’. 
In such cases a sample would have to be much larger than the available 
population before any significant inferences could be made about rela- 
tionships between the variables. An example is a study of science parks 
in the UK (Massey, Quintas and Wield, 1992), where at the time of the 
study there were only 38 entities fulfilling the definition of science park 
used in the study. Although the study obtained data on the whole popu- 
lation which helped to describe the extent of the phenomenon, the small 
numbers meant that when it came to results like ‘the parks in the south 
are more developed’, the authors noted that ‘no serious conclusions can 
be drawn from these figures’ although they might ‘point to an important 
line of enquiry’. Data that might have been obtained from a sample of 
parks would have been even harder to interpret except as data about 
particular cases. 


The above points are all reasons why the logic of defining a sample in a 
survey method is not generally applicable to choosing and framing case 
studies. In fact even when it might be feasible to define a representative 
sample, the logic of the case study approach is quite different, the choice of 
cases being based on the idea of challenge. 


If cases are to be challenging, they must have a certain relation to the ideas 
behind existing or proposed policies and actions. There are different possi- 
ble relationships between the cases chosen and the ideas, corresponding to 
different strategies for choosing case studies. These can be broadly grouped 
under three headings: exploratory, single case, and multiple case. The next 
sections look at each of these in turn. As we shall see, there are ways in 
which it is possible to choose single or multiple cases in order to make a 
general argument without resort to the concept of representativeness. 
Example 14.2 is a practical account of using a case study approach for 
research on NGO influence on environmental policy, including why the 
idea of a representative sample was rejected in favour of using multiple 
case studies in a particular combination. 


316 


14 CHALLENGING CASES 


EXAMPLE 14.2 A STUDY OF NGO INFLUENCE ON 
ENVIRONMENTAL POLICY 


Between 1993 and 1996 I was part of a research team studying NGO 
influence on environmental policy. We were concerned that the study 
should include the relationship between local policy and policy influ- 
ence, in Asia and Africa as well as in the ‘North’, and the issues on 
global environmental agendas. After some debate we formulated the 
following research question: 


Why are NGOs influential in affecting the development of certain 
policies related to global environmental problems?’ 


(Potter, 1996, p. 1) 


This wording was chosen because we wanted to focus on the reasons 
for successful influence in certain cases without implying that NGOs 
are necessarily influential in general or in all circumstances. We then 
set about structuring the research in terms of a number of detailed 
case studies, which would allow us to concentrate on how NGOs 
achieved influence in those cases where they did so, and on why they 
were able to do so in some cases and not others. 


Although we wanted a variety of examples in order to achieve a de- 
gree of general applicability for our results, it was soon clear that 
there was no question of representiveness being a criterion for the 
cases we chose to study. First, representative of what? It was impos- 
sible to conceive of a ‘list’ of all the examples of attempted NGO 
influence in order to derive a representative sample of such attempts. 
What exactly is ‘an attempt at influence’, anyway? Presumably a 
clearcut case of a change in policy acknowledged as being caused by 
an NGO campaign would count. What about a long-term change in 
the way an issue is conceived by the general public or by world po- 
litical leaders which may have been partly the result of long-term 
consciousness-raising on the part of various NGOs and others? And 
what about unsuccessful attempts? Quite apart from not being able to 
define these precisely, or even find out when they had occurred, we 
were actually more interested in reasons for success where there was 
success than in any statistical notion of how frequently different cam- 
paign tactics succeeded. 


Then, we could not achieve access to all the campaigns we wanted to 
study, even though we had local partners who undertook research 
with us in most of the 10 countries in which we worked. In some 
cases, the NGOs concerned feared a backlash if they were seen to be 
co-operating with an international research team. In others language 
or distance and expense were barriers. 
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You might suppose that even though a representative sample of influ- 
ence attempts could not be defined, we could have studied a sample 
of NGOs instead. However, there are plenty of problems with this 
approach. In most countries there is no authoritative listing of all 
NGOs. Those organizations registered under national legislation may 
leave out particularly interesting examples. Once a list was compiled 
that cut across national boundaries, we could not be sure that the 
criteria for inclusion were consistent between countries. NGOs ac- 
tive and successful at policy influence are also a minority among the 
mass of small service providing or mutual aid organizations, so, al- 
though we could focus on some of the large, policy-oriented NGOs, 
those studies would not be ‘representative’ in any clearly defined sense. 
Even at this level, NGOs differ in so many ways (size; focus on cli- 
ent, member or public benefit; local, national or international; south- 
ern or northern; cultural basis; etc.) that a sample that allowed com- 
parison between data on all these dimensions would be impossibly 
large and unwieldy. 


Preliminary literature study and interviews with key NGO activists 
in each of the 10 countries gave us a list of potential cases and also 
allowed us to develop some theoretical ideas. In particular, we devel- 
oped some criteria about how NGOs’ influence would depend on a 
number of general factors: the political context (democratization — 
whether it was operating in a state with or without the several fea- 
tures of democracy which we identified); the nature of the target or- 
ganization (how amenable to influence was the agency whose poli- 
cies the NGO was trying to change); and international links and net- 
working. 


We were also able to differentiate between forms of NGO activity — 
what I have called the ‘four Cs’: collaboration; confrontation; com- 
plementary activities; and consciousness-raising (Thomas, 1996b). 
Finally, our preliminary analysis suggested that different circumstances 
and strategies would make each more likely to succeed, the circum- 
stances being defined mostly in terms of political context and target, 
and strategies including crucially whether and how international links 
were used. Hence we arrived at a number of ideas (or ‘hypotheses’) 
such as: ‘Confrontation is most likely to succeed where there are multi- 
party elections and also a diversity of local power centres’; “Interna- 
tional networking generally makes NGO campaigns more likely to 
achieve influence by increasing the political ‘space’ and providing 
extra resources’ (plus others referring to the other ‘Cs’ and the tar- 
get). 


Our choice of case studies was then informed, on the one hand sim- 
ply by what the key activists we interviewed said we would learn a 
lot from, and on the other by seeking cases which would give evi- 
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dence for or against the ‘hypotheses’. In doing this we included ex- 
amples which were anything but representative in the countries con- 
cerned, including one in Nigeria where international networking 
played a prominent role, and the one depicted in Example 14.3 of the 
1990 campaign against the Southern Okavango Integrated Water 
Development Project (SOIWDP) in Botswana where multi-party elec- 
tions were held and the government was confronted and forced to 
change its policy (Thomas, 1998). 


14.2.2 EXPLORATORY STRATEGIES 

This first strategy is aimed at finding as many different types as possible. It 
tends to lead to describing many cases but in not much detail. It corre- 
sponds to the first two of the four aims of case study research given by 
Langrish (see Section 14.1.2 above): developing labels for a classification 
scheme, and looking for principles behind a taxonomy. This kind of explo- 
ration is particularly appropriate when there is no prior way of defining a 
‘population’ of which the cases are members, and it may be followed by 
studying one or a small number of cases in much more detail. 


Searching for cases of different kinds has to start somewhere, so you will 
generally have to use some kind of preliminary typology (e.g. Langrish 
suggests looking at small, medium and large firms in different industrial 
sectors; in Example 14.1 we used different modes of start-up to look for 
different types of co-operative). However, you are actually looking for as 
many types as possible in a rather different sense — which will depend on 
how the research relates to policy and to the ideas behind the policies you 
are concerned with. 


In Section 14.1.2 we also noted that different types of cases come from 
different answers to ‘How?’ questions. Thus you are probably looking for 
different types in terms of different answers to questions like ‘How do 
firms succeed?’, i.e. different strategies for succeeding or different ways of 
surviving. It often happens that your preliminary typology does not really 
correspond to differences defined in this way. With an exploratory strat- 
egy, you are basically looking for as many different answers to your ‘How?’ 
question as possible, but by the end of it you may have to reconceptualize 
and formulate a new typology, as we did in Example 14.1 with our marginals, 
radical marginals, etc. Sometimes, if it is a question of an exploratory phase 
followed by more detailed case studies, you may even reconceptualize in 
this way at a point well into the later phase of the case study research. 


Given that the basic idea is to look for as many different types of case as 
possible, then you might start with cases which are easily accessible and 
then use a number of heuristic devices to extend your search. 


1 Use obvious differences (age, size, geography, etc.) to form a prelimi- 
nary typology. 
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In Example 14.2 we chose a small number of different environmental 
issues and countries as a starting-point. In Abbott’s khannawalli study 
(Chapter 9) she started by looking for cases of different caste and mari- 
tal status. By this means you can soon be looking for at least one case to 
fit in each of the boxes of a small matrix (e.g. an example each of a 
small, medium and large firm in manufacturing and non-manufacturing 
would mean finding 3 x 2 = 6 examples). 


Tell ‘experts’ about the cases you are already looking at and ask them to 
suggest other cases which might uncover different types. 


In Example 14.2, we started with a background literature study and a 
number of interviews in each country in the study with NGO activists, 
in order to gain a general understanding of the political context and the 
role of NGOs. Treating the activists as experts, we asked each of them 
to suggest campaigns or other attempts at influence that they thought 
we might learn from. 


Use multiple sources (e.g. if you obtained cases from a list of members 
affiliated to a certain group, or firms in a federation, look for a different 
group or federation or another source altogether to find non-members 
who may differ in unpredictable ways from the first set of cases you 
obtained). 

Abbott (Chapter 9) originally obtained access to khannawallis who were 
members of the AMM; she later deliberately looked for khannawallis 
who were not members of that organization. 


Ask people involved in one case about the existence of others, particu- 
larly if they can think of cases where the strategy or the way of trying to 
succeed is quite different. 


A simple example of this was the way Abbott (Chapter 9) asked 
khannawallis who were AMM members for names of others outside the 
association. 


Think of logical alternatives. 


If all the cases you have found seem to do a particular thing in a certain 
way, look for examples where that thing is done in a different or even 
opposite way. In Example 14.2 we found a view in many African NGOs 
that they were obliged to collaborate with government because of po- 
litical necessity and cultural expectations. So we deliberately sought 
cases where African NGOs had confronted government, even though 
they were few and far between. 


Finally, look in different social and political environments. 


Taking Langrish’s analogy of case studies as animals, and thinking of 
how you would look for new kinds of animal on a newly discovered 
island, you might look on riverbanks, on the shoreline, in woods, in 
wetlands as well as in open country. Similarly one would look for dif- 
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ferent kinds of survival strategy among firms in sectors characterized 
by different competitive environments. In Example 14.2 we looked for 
NGO campaigns in countries with more or less democratic regimes, 
expecting the strategies for achieving influence to be very different. 


14.2.3 SINGLE CASE STRATEGIES 

The point of choosing a single case to study is to explain how and why 
something happens by looking in some detail at the interrelationships in- 
volved and the inner workings of the case to be studied. 


Yin (1994) suggests three main rationales for a single case design: 


* the critical case (which may be a test of theory or a way of comparing 
theories or ideas); 


* the extreme (sometimes unique) case; and 


* the revelatory case (where the researcher gains access to a previously 
unavailable situation). 


Critical cases may be set up rather like experiments, with conditions known 
to be favourable, or they may be chosen by repute. The latter method is 
similar to the idea of choosing a case of ‘good practice’, which is also one 
of the basic types of case study selection suggested by Langrish. In his 
types of selection he also includes two very pragmatic ones, namely ‘the 
one next door’ and the ‘cor, look at that!’ — though the latter is effectively 
the same as Yin’s ‘extreme’ case. 


Any of these methods can be used to choose a case to study in detail, and 
may be appropriate depending on the circumstances. If a small number of 
cases is required, similar rationales can be used for each, although the choice 
of a particular combination of cases will be subject to the strategies for 
multiple cases described in the next section. First, let us look in a little 
more detail at each of the suggested rationales for choosing single cases, 
starting with the pragmatic ones. 


1 The one next door 


If you are interested in the detailed workings of a case, and there is no a 
priori reason to differentiate one from another, then there is no reason 
not to choose on a completely pragmatic basis. Why not study a case to 
which you have easy access? This might mean, for example, studying a 
case which involves an agency that is commissioning the research, al- 
though there are obvious dangers of bias to be avoided in such an exam- 
ple. 


2 ‘Cor, look at that!’ (the extreme case) 


Unusual or unique cases can be particularly useful for showing up causal 
mechanisms. For example, the research on NGOs and environmental 
influence (Example 14.2) included Botswana, where there is relatively 
little NGO activity and most NGOs work collaboratively with govern- 


321 


THINKING WITH DATA. 


ment but are rather uninfluential in policy. We deliberately chose to 
study one of the very few cases where NGOs had clearly opposed gov- 
ernment and forced a policy change (Example 14.3 below). 


The revelatory case 


Perhaps the most useful examples of a situation which suddenly be- 
comes available for study relate to new phenomena. In the study of 
science parks mentioned in Section 14.2.1 (Massey, Quintas and Wield, 
1992), these parks were in general a fairly new idea so that getting real 
data to explain how they evolved, as opposed to ideas on how they 
should evolve, required studying one or a small number of the few cases 
that had been going long enough to reveal how they worked in practice. 


The critical case (1): a quasi-experiment 

This is the first of three ways of setting up a case study so as directly to 
challenge theory or ideas in order to explain how the cases work. In an 
experiment you control the starting conditions and see if the outcome 
expected by your theory does in fact occur. Here you cannot control the 
starting conditions but you can choose a case to match pre-determined 
characteristics and follow it to see if it reaches the theoretically pre- 
dicted outcome. For example, in looking at the impact of a policy you 
might choose a case such that, according to the ideas behind the policy 
you are investigating, conditions are as favourable as possible for a 
good outcome. 

Another kind of example is given by Robinson’s (1998) study of NGOs 
in health care reform in Tanzania. She suggested a way of conceptualiz- 
ing NGOs not purely as service providers or advocates but in terms of 
public action, in which they would be expected to play a role in defin- 
ing what is to be regarded as ‘public need’ as well as help to meet that 
need alongside other types of agency. She chose a case study of the role 
of NGOs in community-based health care because it seemed to be an 
example where the conditions were such that the public action role of 
NGOs should be clear. 


The critical case (2): chosen by repute 

The second way of deliberately challenging theoretical ideas is to choose 
cases reputed to be cases of ‘good practice’ or where there has been a 
‘success’. Here you would be checking whether the reasons for ‘suc- 
cess’ or for the reputation of ‘good practice’ correspond to your ideas 
for what should be expected to lead to success. For example, in a fol- 
low-up to the study of co-operatives in Example 14.1, you could pick a 
well-known example of a radical bookshop (clearly a ‘radical marginal’) 
and look in detail at whether the mechanisms by which it maintained 
itself were indeed those suggested by the ideas behind the notion of 
‘radical marginal’, including high labour turnover as well as moral 
motivations and political commitment. 
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6 The critical case (3): comparing two sets of ideas 

This involves framing a case so that two competing sets of ideas or 
theories can challenge each other. This is perhaps the most testing of all 
single case designs, corresponding to the ‘contrastive approach’ to data 
analysis put forward by Mukherjee and Wuyts in Chapter 11. The in- 
vestigation will look for evidence of which set of answers to the ‘How?’ 
and ‘Why?’ questions provides the best story or explanation. In Exam- 
ple 14.3 below there were two competing theoretical explanations for 
why the Botswana government backed down: one seeing it as a response 
of a democratic government to views of a section of its population which 
NGOs had helped to articulate; the other attributing it to the power of 
international networking among NGOs. As you will see when you look 
at the Example 14.3, neither explanation was exactly correct by itself, 
but having more than one ‘handle’ on the case allowed us to gain some 
very useful insights. 


14.2.4 MULTIPLE CASE STRATEGIES 

It is an often repeated fallacy that you cannot generalize from case studies. 
It is quite possible for general conclusions to be reached from a relatively 
small number of case studies — or even from one case study. In fact the 
main point of multiple case studies is to challenge the theoretical ideas 
which have been developed so that the ensuing generalization can be that 
much stronger. Either the case studies will confirm your ideas so that you 
are more confident about generalizing on their basis or else the case stud- 
ies will throw doubt on the applicability of your ideas in different circum- 
stances, in which case you should be able to reconceptualize and general- 
ize on the basis of changed or new ideas. 


What you can not do on the basis of case studies is statistical generaliza- 
tion. Since you are not working from a statistical sample you cannot make 
general descriptive statements about the whole of a population. Abbott's 
study of khannawallis (Chapter 9) could not be used to say what propor- 
tion of them used loans for purposes of consumption rather than investing 
in their productive activities of home cooking. We could not generalize 
from the cases of worker co-operatives we studied (Example 14.1) to esti- 
mate the proportion of co-operatives of each of the types we identified or 
what proportion of each type failed within, say, five years. 


What you can do with case studies is theoretical generalization. If you 
have a theoretical idea about how success is the outcome of the interrela- 
tionship of certain factors or influences for example, and if this idea suc- 
ceeds in explaining your case studies, then you can make general explana- 
tory statements that apply to all cases where the interrelationships are similar. 
Any challenge to your general explanation must be either a claim that the 
case studies were misinterpreted, which would need to be carefully an- 
swered, or else of the form ‘Here’s a case where that doesn’t apply’. The 
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latter should lead to a useful debate about why that case is different from 
the ones you studied and might lead to further reconceptualization, but in 
principle does not invalidate your theoretical generalization. 


We have seen that, in choosing more than one case so that the combination 
challenges your ideas, you are not sampling. Instead, the logic to use is 
similar to that of multiple experiments. Each successive experiment is de- 
signed to reinforce the theory, to extend its application, or to force a re- 
think or modification. As Yin (1994) points out, to be done in a similar way 
case study design has to be theory-led; in other words ideas have to be 
worked out before you can decide which cases to study. Yin suggests bor- 
rowing the principle of replication from experimental method. He discusses 
two main forms of replication: 


* literal replication; and 
* theoretical replication. 


In literal replication the cases are chosen to be as similar as possible to 
each other, according to your idea of what conditions are important. You 
check whether similar processes and outcomes do in fact occur, For maxi- 
mum impact, and to avoid the possibility that your evidence would support 
more than one idea, you may want to choose cases which differ consider- 
ably in ways which your idea expects to be unimportant. If, either from a 
priori reasoning or as an outcome of a preliminary exploratory stage (see 
above), there are several theoretical categories of case, then you may need 
a literal replication of each (i.e. at least two similar cases in each category). 


Abbott’s study of khannawallis included literal replication when she stud- 
ied cases of more than one married woman to check that in each case the 
same type of consideration (caste and kinship links, contribution to mixed 
livelihood strategy, etc.) was behind the way the activity was organized. 
With the study of NGO influence, the case study of the campaign against 
the SOTWDP (mentioned in Example 14.2 and described in Example 14.3 
below) could have been followed by a case study of another confronta- 
tional campaign in Botswana or a similar country where multi-party elec- 
tions are also held, to see if both campaigns succeeded for similar reasons 
as envisaged by our theoretical ideas. 


Theoretical replication means choosing case studies which are different 
from each other in theoretically significant ways. They should be chosen 
such that different or opposite conditions apply, so that your ideas lead you 
to expect different or opposite outcomes. You then investigate whether these 
different or opposite outcomes in fact occur, how, and why. 


An example of theoretical replication from Abbott's khannawalli study is 
the inclusion of women without male partners (deserted or widowed), or 
whose husbands were unemployed, as well as married women with em- 
ployed partners. Assuming that the khannawalli activity is best interpreted 
as contributing to livelihood strategies, one would expect married women 
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to rely less on the activity and for it to vary in importance over time in 
relation to other income possibilities, whereas for deserted or widowed 
women it is more likely to be of prime importance as a livelihood strategy. 
This theoretical replication allowed Abbott to check that the different out- 
comes did more or less correspond to the differences in circumstances and 
thus tended to confirm these ideas. 


With respect to the NGO influence study, another of the cases studied was 
of how NGOs in Botswana related to the government policy of fencing 
communal lands. The NGOs concemed opposed this policy in principle 
but certainly did not pursue any confrontation with government and were 
completely ineffective in achieving any influence (our Botswana research 
partner’s report on the case was entitled ‘Silence of the lambs!’ (Selolwane, 
1998)). 


Note that all three versions of the ‘critical case’ can be used in replication. 
Perhaps the last is the most powerful. Contrastive inference (see also 
Section 14.3.2) — comparing two sets of competing ideas, in similar and 
then theoretically dissimilar cases, is a very good way of ensuring that 
your case studies remain challenging. This approach was being followed 
both in the khannawalli study (where the notion of the activity as a micro- 
enterprise was constantly being contrasted with the livelihood strategies 
idea) and, as we will see, in Example 14.3 below. 


14.3 CHALLENGING A CASE WITH YOUR IDEAS 


This section is concerned with how to conduct a case study investigation. 
Just as data will not tell a story unless you ‘fire questions’ at them (Chapter 
11) so you cannot expect to get an explanation from a case study unless 
you challenge it with your ideas. Thus the approach suggested is based on 
the notion of developing your own theoretical ideas in advance, so that you 
know what kind of story or explanation to expect, and then using these 
ideas to give a focus and edge to the investigation. 


In what follows we look first at the methods which may be combined in 
carrying out a case study and then at ways of trying to achieve rigour so as 
to be able to defend your results. Example 14.3 is a practical illustration of 
many of the points made. 


14.3.1 METHODS TO USE IN A CASE STUDY INVESTIGATION 

As noted in Section 14.1.3, a case study investigation is really a frame- 
work for research rather than a specific ‘method’. Within that framework 
you can use any combination of the methods which have been discussed in 
the other chapters of this book, including ‘thinking with’ paper, people, 
organizations, and data. 


What follows are brief notes about how you might use different methods 
within a single case study. 
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Background literature and interviews 

It is important to hone your ideas about the ‘How?’ and “Why?” questions 
you want to ask before getting in to the detailed investigation. You will 
need to do a small literature study as part of any case study and you should 
probably start this before making direct contact and requesting access. This 
applies particularly to collecting and analysing the most easily available 
‘grey’ literature (Chapter 5). If the case is about an organization, for exam- 
ple, you should obtain annual reports, publicity material, recent newslet- 
ters and other publicly available documents from the organization. An- 
other useful preliminary is open-ended interviews with peripherally involved 
‘experts’ — people you can talk to safely even before you know if this is a 
case to which you will be able to get good enough access. This is another 
example of Hanlon’s* finding the woman who knows’ from Chapter 3. 


Access 

Your study may well have been preceded by an exploratory phase of find- 
ing out at a much lower level of detail about a relatively large number of 
cases of different types. While contacting those involved you may have 
been able to arrange in principle that you could come back and do a more 
detailed study. Or the question of access to the particular case or cases you 
want to study in detail may be more tricky. Depending on the circumstances, 
you may need a ‘gatekeeper’ (Chapter 6). For example, studying cases of 
the ‘objects’ of a policy may require the policy-making agency (which may 
be your own, of course) to agree to facilitate access to those ‘objects’, as 
with Abbott gaining access to khannawallis through the AMM. Alterna- 
tively, if the case is one involving interactions between several individuals 
or organizations you may find access to some of them easy, others requir- 
ing a gatekeeper and yet others harder or impossible. 


Following through in focused interviews 

Much case study analysis involves detailed interviewing (‘focused inter- 
views’ in the terms of Chapter 6). A key element of such interviews is not 
only to have some standard headings but also to follow through. When 
asking someone how or why certain things happened, it is likely that new 
elements and contradictory causes are suggested, in which case you should 
follow through on investigating chains of causality. This may mean new 
people to interview (whose importance to the case you hadn’t realized) and 
whom you now have to trace. It also means asking different interviewees 
to explain answers given by others which seem at odds with their own 
accounts or which leave loose ends in the explanation you are building up. 
Following up reasons behind reasons given is much more important in this 
type of interviewing than covering the same topics in each interview. 


Using data analysis or survey within the case study framework 

Although focused interviews are usually the main method used, there is 
often a place for other methods within the case study framework. Particu- 
larly with large single cases, there may well be definable statistical 
populations within the boundaries of the case, on a sample of which a struc- 
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tured survey could usefully be carried out; or there may be official data 
available on such a population that could be analysed. An example from 
the science parks research (Massey, Quintas and Wield, 1992) is data on 
the individual firms within the parks studied. You would have to avoid 
what Mukherjee and Wuyts (Chapter 11) call ‘hoarding data’ — padding the 
report on your case study with numbers and tables without using them. Be 
sure to ‘fire questions’ at such data in such a way that the answers throw 
light on the main ideas you are investigating in the case. 


Building a story 

The main point of a case study is to put together a coherent explanation, or 
story. In the first place, you are trying to build a story based on your origi- 
nal set of ideas. You identify the major actors and get their stories, to see if 
an overall version emerges in the form expected. You compare this with 
the story that emerges from other evidence such as quantitative data. Much 
of what Hanlon says in Chapter 3 applies directly to gathering evidence in 
a single case study, particularly the sequence of ‘throw the net out wider’, 
‘redefine the question and select the key issue’ and finally ‘discard and 
deepen’. In particular, this means that building a story may mean 
reconceptualizing, and your original ideas may end up not only challenged 
but changed as a result. 


EXAMPLE 14.3 COMBINING METHODS TO CHALLENGE THE SOUTHERN 
OKAVANGO INTEGRATED WATER DEVELOPMENT PROJECT 


One of the case studies in the research on NGO influence outlined in 
Example 14.2 was a study of the 1990 campaign against the Southern 
Okavango Integrated Water Development Project, Botswana (Tho- 
mas, 1998). This was a government project which would have dredged 
part of one of the main channels of the Okavango Delta and built a 
number of dams in order to supply domestic piped water as well as 
water for irrigation and a diamond mine. It caused controversy be- 
cause in a largely arid country the Okavango is not only one of the 
few sources of water but its delta is also one of the world’s largest 
wetlands, ecologically unique and a big resource for wildlife and in- 
ternational tourism, as well as providing the resource base for some 
100 000 local inhabitants. The project was shelved after opposition 
both from local communities led by a local community-based NGO 
Tshomorelo Okavango Conservation Trust (TOCT) and from interna- 
tional conservationists and NGOs, including Greenpeace. 


This case study was chosen because it potentially offered evidence on 
two of the ideas being developed by the research team. The first was 
that a confrontational approach by NGOs is more likely to work where 
genuine multi-party elections are held and where there are alternative 
local power centres. TOCT was unusual for an NGO in Botswana in 
its confrontational stance and indeed was formed by a split from a 
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more longstanding national NGO, the Kalahari Conservation Society 
(KCS), which followed a collaborative strategy including sitting on 
government commissions and quiet attempts at lobbying. We wanted 
to find out why TOCT’s campaign had succeeded. Was it through 
utilizing the democratic processes of the Botswana state, including 
the government's wish to avoid an issue blowing up into an electoral 
weapon for its opponents? Did they exploit the existence of ‘alterna- 
tive power centres’, such as local MPs and councils of the opposition 
party as well as the ‘traditional authorities’ and chiefs with important 
influence both locally and nationally? If so, our original idea would 
be considerably strengthened. 


The other idea on which we wanted evidence was that NGOs improve 
their influence on policy through engaging in international NGO alli- 
ances. Some of the early grey materials which we collected were re- 
ports and pamphlets from South African conservationist sources, which 
claimed that TOCT had worked together with Greenpeace and forced 
the Botswana government to back down through a combination of 
Greenpeace threats (such as an international “Diamonds are for Death’ 
campaign or an EU boycott of Botswana beef) with mobilization of 
the local population by TOCT. Had TOCT actually been more effec- 
tive because of an alliance with Greenpeace and other international 
NGOs? 


In following up this case, we interviewed protagonists representing 
various interests. Here it was useful that I worked on the case jointly 
with a local colleague from Botswana. She was able to interview lo- 
cal and national politicians, high-ranking officials including those who 
had held key positions at the time of this campaign but since moved 
or retired, and directors both of the state diamond corporation and 
NGOs such as KCS. In the UK I interviewed one of the Greenpeace 
team which had spent three weeks in Botswana in 1990 investigating 
the project, and I also visited the locality and spoke to conservation- 
ists, founding members of TOCT and someone who had been a ‘whis- 
tle-blower’ within the government's water department. We found many 
discrepancies in their accounts, notably one between the South Afri- 
can sources noted above and the claim from Greenpeace that they had 
never threatened any international campaigns against the Botswana 
government and their study tour had been on that government’s invi- 
tation, culminating in a joint communiqué and press conference. 


We also used different types of evidence, including not only inter- 
views and grey materials but also media reports from the time, con- 
sultants’ reports on the design and environmental assessment of the 
project, internal memos from government departments which some 
of the interviewees were prepared to let us see, a video of a local 
community meeting which had attracted several hundred people, and 
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at which government officials had been harangued for several hours, 
and Greenpeace archives. These allowed us to cross-check some of 
the discrepancies and in some cases we went back and asked inter- 
viewees supplementary questions or asked them to comment on a 
draft of our case study report. In the end where accounts differed we 
had to make informed judgements on what was likely to have actu- 
ally occurred and why. Thus, although we found several media re- 
ports from the time repeating the story about Greenpeace threats, we 
concluded that probably they had never actually made such threats 
but certain local and South African conservationists may have ‘placed’ 
stories which talked of the ‘possibility’ of a Greenpeace campaign. 


On the first of the ideas we were testing, it seemed to be a concern to 
appear democratic to international observers as much as the actual 
workings of democracy which led the Botswana government to back 
down. On the second, the case helped us develop the idea that it may 
be the internationalization of the issue rather than specific alliances 
which helps local NGOs to win a battle. Too much reliance on the 
involvement of NGOs from outside a country may lead to a potential 
backlash — in this instance Greenpeace is now often accused of inter- 
ference and their involvement invoked in some circles to deride envi- 
ronmentalist concerns. Thus in the end, the explanation we gave 
modified considerably both our original ideas, particularly that on 
international NGO alliances. 


14.3.2 ACHIEVING RIGOUR IN A CASE STUDY INVESTIGATION 

Case study methods present a problem with respect to rigour. It is easy to 
be criticized for simply finding evidence to fit your preconceived ideas. 
Since you choose the case to study and start with a theoretical explanation 
already in mind, then finding evidence to fit your pre-prepared story may 
appear to be a self-fulfilling prophesy, Case study research can indeed be a 
matter of using a mixture of mainly qualitative methods and subjective 
judgement to tell a story which confirms what you already worked out was 
to be expected. Yin (1994) argues that this is unavoidable but acceptable. 
The important point is to be open to the possibility that the story is wrong 
or needs changing. If your ideas really are open to challenge then simply 
showing that as more and more detail comes to light the same basic expla- 
nation remains consistent is in itself a useful result. 


Some of the ways of showing that your ideas really are open to challenge 
are similar to the ways rigour is achieved in semi-structured interviews 
more generally (not surprising since as we have seen such interviews usu- 
ally form a big part of any case study investigation). From Chapter 6 you 
will recall that Philip Woodhouse suggests triangulation and documenta- 
tion as the two key means of achieving rigour in semi-structured inter- 
views, and these are important for case study research. 
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Triangulation is getting evidence on the same point from different points 
of view. It means both using different methods of investigation (analysing 
archive records, secondary data or media reports as well as published lit- 
erature and interviews) and getting evidence from several sources (inter- 
viewing several individuals; checking reports from several agencies). A sim- 
ple example is Abbott’s interviews with clients as well as khannawallis 
themselves (Chapter 9). Another, from Example 14.1, was checking 
co-operative members’ accounts of the history and performance of their 
co-operative against records such as minutes of meetings and financial 
accounts. 


Documentation: Since there is so much subjective judgement in carrying 
out a case study investigation, it is essential to keep good records of the 
progress of the study. This should enable you, either in response to criti- 
cism or for your own purposes, to go back and check what evidence actu- 
ally led you to frame your explanation in the way you eventually did. 
Ideally these records should include the following: a diary; notes on inter- 
views (including headings worked out in advance, additional lines of ques- 
tioning, suggestions for further investigation and loose ends thrown up); 
documents obtained (with their source, notes on your skim reading — see 
Chapter 5 — perhaps more detailed notes later, and again loose ends to be 
followed up elsewhere); other data (with how obtained and results of analy- 
sis); and other items as required in your particular enquiry. 


An additional, and very important, way of avoiding the accusation of the 
self-fulfilling prophesy is to follow a contrastive approach similar to that 
taken in the case of data analysis using ‘contrastive inference’ by Mukherjee 
and Wuyts in Chapter 11. 


This means, again in the words of Mukherjee and Wuyts, using more than 
one ‘handle’. In other words, you start with more than one set of competing 
ideas; you have more than one type of explanation in mind and you use the 
evidence from the case study to decide which explanation stands up best. 
(This is like the third version of the ‘critical case’ in Section 14.2.4) 


Yin gives the example of Allison’s (1971) study of the Cuban missile crisis: 


Allison posits three competing theories or models to explain the 
course of events, including answers to three questions: why the 
Soviet Union placed offensive (and not merely defensive) missiles 
in Cuba in the first place, why the United States responded with a 
blockade (and not an air strike or invasion), and why the Soviet 
Union eventually withdrew the missiles. By comparing each theory 
with the actual course of events, Allison develops the best 
explanation for this type of crisis. Allison suggests that this 
explanation is applicable to other situations, thereby extending the 
usefulness of his single-case study. 


(Yin, 1994, p. 5) 
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Throughout the research on worker co-operatives (Example 14.1) we were 
using case studies to find evidence both for the suggestion that mode of 
start-up remained the greatest influence on a co-operative’s performance 
and the contrasting notion that it is how commitment (personal and finan- 
cial) is maintained that determines the ways that a co-operative could sur- 
vive and flourish. 


Example 14.3 also shows how contrastive inference worked in a rather 
different context. There we found different informants telling contrasting 
stories, some attributing the change of government policy to its democratic 
nature, others to the internationalization of the campaign. The explanation 
we ended up with was rather more complex than either of the simple start- 
ing hypotheses. 


14.3.3 SUBJECTIVE JUDGEMENT, VALIDITY AND PRESENTATION 

One can expect competing stories or explanations based on evidence from 
different sources. To some extent triangulation will eliminate some of the 
differences, but it is unlikely that there will be a single clear version agreed 
by all in every respect. There is a certain amount of professional judge- 
ment involved in putting together what others would accept as a rigorous 
and valid interpretation of a case — which could be used as evidence for or 
against your original ideas or in favour of a newly modified version. 


14.4 PRESENTING CASE STUDIES AS EVIDENCE 


What are you to make of the evidence from case studies in order to present 
the results and try to make an impact on policy? It is useful to think in 
terms of a two-way interchange between you as investigator with your 
ideas, on the one hand, and the case study and the evidence it produces on 
the other. 


There are two extreme types of possible result. First, your original ideas 
may stand up well but challenge the cases such that new insights arise 
which are something of a surprise to those involved in the cases them- 
selves. This was more or less what happened in the khannawalli study 
where Abbott’s idea was that the notion of contributing to mixed liveli- 
hoods strategies would explain the activity and this idea stood up better 
than the alternative throughout despite challenge from a variety of cases. 
This was to some extent a challenge in turn to the AMM, whose policies of 
lending to khannawallis as though to micro-enterprises looked less securely 
based as a result of the study. 


At the other extreme, if the evidence from the cases investigated consti- 
tutes a sufficient challenge to your ideas there may be a need to stand back 
and completely reconceptualize. Example 14.1 explained how something 
like this occurred in the case of the research on worker co-operatives in the 
UK. In the science park research referred to in Sections 14.2.1 and 14.2.3 
above there was a point where the original conceptualization which at- 
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tempted to explain science parks as attempts to create synergy between 
university-based intellectuals and innovators and entrepreneurs had to be 
dropped. In the new version, property development and regional develop- 
ment considerations, which differed considerably between more central 
and more peripheral parts of the UK, came to the fore in explaining how 
and why different science parks prospered in different ways. 


Between the two extremes are various possibilities: for example, one’s ideas 
could lead to a slightly modified view of the cases while in turn they could 
be somewhat modified as a result of challenge from the cases. More than 
one set of ideas could have been pitted against each other, and unless one 
or other falls down completely this could mean a result with two compet- 
ing overall explanations each of which provides some evidence. Example 
14.3 showed how the final explanation can well owe something to both of 
the originally competing ideas. 


How results are to be presented in each of these situations is of course 
itself part of the policy process. If the evidence from your case studies is to 
have any impact you have to consider its presentation, how the case stud- 
ies are used as evidence, which parties will be challenged by the findings, 
and so on. All the considerations in Chapter 10 on ‘communicating results” 
will be important to ensure that the challenge of the cases you have chosen 
and analysed is not limited to your own thinking but makes a real differ- 
ence. 


332 


CONCLUSION: PERSONAL EFFECTIVENESS 
AND INTEGRITY 


ALAN THOMAS AND JOANNA CHATAWAY 


The Introduction to this book started by noting that it is about ‘research to 
inform policy and public action’. Hopefully the chapters of the book will 
have helped to make you more effective, whether as a policy researcher or 
as a user of policy-oriented research. It should also have become clear how 
important it is to maintain standards of personal integrity in the face of 
some difficult ethical issues which can arise in policy-oriented investiga- 
tions. Indeed, we would maintain that in the long run integrity is the key to 
effectiveness. 


But what exactly is ‘effective’ policy research? Does ‘effective’ mean get- 
ting a result in terms of action, policy implementation or change? If so, 
what if things don’t work out as expected? Can the research still be re- 
garded as effective because it led to action? On the other hand, what if 
research helps to clarify the consequences of a policy which works well 
and has well-understood negative consequences for some groups or inter- 
ests? Is this effective simply because it has led to better understanding, 
irrespective of whether or how that understanding is then used? Or is such 
research to be judged in terms of more or less desirable outcomes: effec- 
tive if it has helped less powerful groups to improve their position at the 
expense of the more powerful, but not effective if the research was at the 
service of those who exploit? 


1 DIFFERENCES BETWEEN AUTHORS 


We noted in the Introduction the difference between those who think in 
terms of ‘discovering reality’ and those for whom there may be not one but 
several simultaneously valid versions of reality from different viewpoints. 
While both groups may want to see the results of their research used for 
policy or action, the former will judge the research ultimately by how clearly 
it uncovers how things really are, whereas for the latter the only way to 
judge research is by its ‘utility’. 
Indeed, different authors in the book do appear to take different positions 
in this respect. Some clearly hold to the idea that there is an underlying 
reality which research can find out more and more about, thus improving 
understanding and creating more and more certainty in knowing the results 
of policy change. For example, Mukherjee and Wuyts write: 

Policy analysis seeks to understand reality, with the explicit pur- 

pose of informing public action aimed at changing this reality. 

(Chapter 11, p. 239) 


Other authors, however, point out how different accounts or pictures of 
reality are simultaneously possible based on different perspectives and in- 
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terests. Blackmore and Ison, for example, show graphically in Figure 2.2 
how different actors see different systems with different boundaries in the 
same situation. So how is one to choose between perspectives, except on a 
subjective basis of what seems most useful? Hanlon in Chapter 3 specifi- 
cally suggests redefining the question one has been asked to investigate in 
such a way that the new conceptualization is more likely to bring about 
what the investigator sees as a useful result but is still acceptable to the 
agency commissioning the research. 


This is not the only point of disagreement between authors. In Chapter 3 
Hanlon also suggests using the short cut of ‘finding the woman who knows’, 
while O’Laughlin in Chapter 5 warns against following short-term con- 
sultants whose ‘quick and dirty’ methods include over-reliance on ‘gos- 
sip’. In Chapter 13, Mayer bases her whole discussion on how to think 
about and use data when you are trying to convince others to do as you 
suggest, whereas in Chapter 11 Mukherjee and Wuyts argue that, even if 
your main purpose is advocacy of your particular point of view, you should 
nevertheless explicitly contrast rival ideas with yours to see which provide 
the better explanation of the data. 


There are also differences over what is meant by uncertainty, about whether 
participation and empowerment should always be an aim of research, and 
about the desirability of the ‘fast’ in ‘finding out fast’. 


To what extent do these differences all relate to the basic difference noted 
in the Introduction between ‘the realist notion of underlying truth which 
aims progressively to discover by building up knowledge’ and ‘the social 
constructionist conception of the possible simultaneous validity of differ- 
ent accounts or constructions of human interaction based on different per- 
spectives, interests and values’? Or are some of them differences in stand- 
ards, and thus relate to the question of integrity? To what extent are these 
differences reconcilable? 


2 COMMON GROUND 


We will return later to the question of reconciling the differences between 
the authors in this book. For the moment, let us note that, despite these 
differences, on the whole there is a lot of common ground between them. 
Much of this stems from a common interest in investigation to inform de- 
velopment policy and public action. In different ways, the authors follow 
the kind of approach outlined in Sections 4 and 5 of the Introduction. Per- 
haps the strongest commonality is a recognition that policy and action has 
to be thought of in terms of process. The challenge for any policy investi- 
gation, then, is to fit into this process in an effective way. 


POLICY AS PROCESS NOT PRESCRIPTION 
‘Policy as process’ contrasts with ‘policy as prescription’. This is why it is 
worth re-emphasizing the importance of thinking of policy as process. A 
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‘policy as prescription’ view would separate the policy-maker from the 
field of policy and see policy research as finding out about the field of 
policy in order to help the policy-maker prescribe what should be done. In 
terms of Table 1.1 on p. 22, it would concentrate only on the bottom right 
box in the matrix. An example might be agricultural research demonstrat- 
ing the suitability of a particular drought-resistant crop in a certain area. 
Typically such research would not pay specific attention to who would act 
or how the prescribed policy would be implemented. There might be an 
unstated assumption that the government should act or that people should 
simply adopt a new practice because the research had demonstrated its 
superiority. 


The ‘policy as process’ view differs in that the process idea is felt in sev- 
eral ways. I will mention three here. Examples of all of them are found in 
various chapters of the book. 


First, there is a constant process of change in the field of policy. Thus the 
suitability of a crop in a particular area will be changing, not only with 
climate changes but perhaps more importantly in the short term with changes 
in availability of labour, irrigation and other inputs, with changes in taste 
and marketability, with changes in combinations of other crops that are 
habitually grown, and with changes in sources of non-farm income and 
migration, land tenure patterns and so on. This implies that a policy inves- 
tigation has to look for how and why people act as they do, and hence how 
changes occur. In other words it has to analyse social processes, rather than 
just describe the current situation. Hence, for example, it was suggested 
with respect to UK transport policy in Chapter | that investigation needs to 
find out why people make trips in order to understand changes in demand 
and also perhaps to be able to influence them. 


Second, policy development is a process itself. It is not just a question of 
taking the recommendations from an investigation and implementing them. 
There are often several interests and considerable political interaction within 
just one agency. There will almost certainly be policy in the area already, 
to be reinforced or modified, rather than a new policy simply arriving fully 
formed. Thus the promotion of a new drought-resistant crop might have to 
be done by extension agents who already have views and practices involv- 
ing promoting other crops. Any investigation will in practice be just one 
input into a continuing process of definition and redefinition of policy. 
Indeed, I don’t think there is an example in any chapter of the results of an 
investigation being turned directly into a new prescription. The clearest 
examples of a process of interaction between an agency making and re- 
making policy and those doing research for it are perhaps in Chapters 3 
and 10, but this aspect of process is found throughout the book. 


The third area of process is the relation between action and those acted on, 
or between policy and the ‘field’ of policy in which action takes place. 
Rather than the policy-maker being the only active agent and those in the 
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field being passive recipients of policy, in practice many agencies and in- 
dividuals are actively pursuing their different policies and reacting to each 
other, all the time. An investigation is more than just one input to a process 
of policy-making within a particular agency. It is also an input into this 
process of interaction which might affect the actions and reactions of sev- 
eral actors. So although farmers do not simply adopt a new crop because 
research says so, new training for extension agents based on new findings 
is a factor in their decisions, as is locally disseminated knowledge of agri- 
cultural trials carried out on neighbouring farms. Seed companies may be- 
gin to promote the new seeds and others may look for ways of marketing 
the produce. In Chapter 8 the Oxfam-commissioned report on the impact 
of user health care fees in Zimbabwe is described as causing a ‘furore’ and 
in the end both the World Bank and Government of Zimbabwe modified 
their policies. All policy investigation is subject to these considerations. 
However, it is research aimed at changing the way an issue is thought of by 
a wide range of actors rather than at informing or pressurizing one or two 
specific policy-makers that fits with this aspect of process most closely. 
Examples from the book include several cited in Chapter 13 as well as the 
research on UK science parks mentioned in Chapter 14 and that on support 
to Zimbabwean co-operatives in Chapter 3. 


Thus taking a ‘policy as process’ view means policy research can investi- 
gate policy itself as well as the field of policy. In other words, it can range 
over any or all of the boxes in the matrix of Table 1.1. In fact, it may be 
hard to fix boundaries between the boxes, to distinguish policy actors from 
those acted on, and it may be unhelpful to try to do so. Policy research 
means analysing social processes, and seeing public action as resulting 
from existing structures and institutions but also acting upon them and 
thereby transforming them, thus changing the context of policy, sometimes 
in unexpected or unintended directions. 


One investigation does not have to cover everything, however. Not every 
investigation can look at the whole policy process and all those involved. 
An investigation needs to be ‘boundaried’ so as to be effective in inform- 
ing and providing an intervention into the policy process within the given 
constraints of resource and time. Although we may not be able to draw a 
clear line between policy itself and the field of policy, a particular investi- 
gation may focus more on one than the other. 


Let us look briefly at anumber of other points of commonality between the 
book authors. They all relate to aspects of the ‘policy as process’ view, and 
hence to how to turn a limited investigation into an effective intervention 
into such a process. 


THE TWO SIDES OF PUBLIC ACTION 


In considering investigation which is meant to inform public action, we 
should remember that public action has two complementary aspects. One 
side of public action is simply action to meet public (or collective private) 
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needs. But since what are public needs is inevitably contested, the other 
side of public action is action that helps to define what are to be regarded 
as public needs. 


To inform actions to meet needs requires fairly straightforward investiga- 
tions: measuring the extent of needs, assessing the effectiveness of previ- 
ous policies aimed at meeting those needs, and so on. But to provide evi- 
dence to help in defining areas of public need or issues where the public 
interest is at stake requires more open-ended investigations. In particular, 
you must be willing to reconceptualize, and to frame issues in a way that 
will bring them forcefully into the public arena. 


Chapter 5 described contestation over how the problem of reducing rural 
poverty in Africa should be conceptualized, with some reports treating it as 
a matter of improving smallholder agriculture and others looking at the 
variety of income sources available to rural populations, including migra- 
tion to find wage labour. Further useful research on poverty in particular 
African countries could well aim at defining the dimensions of the prob- 
lem and hence exactly what the need is, rather than assuming a particular 
version of need and trying to measure it, Chapter 13 gives several exam- 
ples of research which aimed at demonstrating the need for action in cer- 
tain areas, such as the Third World Network (TWN) trying to show that 
genetically manipulated organisms (GMOs) can have harmful ecological 
effects. These are also contested areas, where the way the problem is con- 
ceptualized (which partly derives from how it has been investigated) helps 
define how it is regarded as affecting the public interest. In these and simi- 
lar cases, it is important to define research questions and carry out investi- 
gation with the need to make an impact strongly in mind. For example, in 
the case just mentioned, although TWN was concerned about the social 
effects of GMOs, it felt it needed to provide evidence on the more techni- 
cal aspects of their effects because the whole field of GMOs is thought of 
as scientific and technical. ‘Grabbing attention’, in Hanlon’s terms, may 
mean re-defining the question so that the answers are more telling, while 
remaining in the realm of what is acceptable to those for whom the an- 
swers are intended. 


AN AGENCY PERSPECTIVE 

Consideration of how policy is to be carried out and who is to act leads one 
to treat all people and organizations involved in a situation as agents, not 
just ‘objects’ of policy. Hence the interest throughout the book in making 
inferences from what has been done in the name of previous policies by 
various agencies. Chapter 5 deals with analysing institutional discourse, 
Chapter 8 with ‘institutional footprints’ and Chapter 12 with institutional 
accounts. There is also the question of the implications of what might be 
done, maybe as a result of policy investigation, and how any change car- 
ried out will affect different people and organizations. Both policy and 
policy investigation generally take place in a multiple-actor field. There 
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may be not just different views as to how social processes operate but 
different interests according to how people and organizations are likely to 
be affected by change. In other words, policy involves politics. Finding 
out fast is always done in a political context. 


Viewing those involved in any situation as agents applies to those gener- 
ally regarded as acted on as well as to those powerful agencies that have 
definite, worked out policies which they try to implement. A good example 
is provided by the study of the impact of health reforms in Zimbabwe car- 
ried out by Oxfam and described in Chapter 8. This study looked at what 
led the World Bank and the Government of Zimbabwe to implement re- 
forms in a particular way and what might induce them to change as well as 
how people reacted to the reforms. The position of the researcher or re- 
search team is of prime importance. How do they relate to those they are 
investigating, to those who are usually regarded as ‘objects’ of policy and 
to the policy-makers? In the Oxfam study mentioned the researchers clearly 
had to establish ‘rapport’ both with the individual users of health services 
that they chose as case studies and with decision-makers in the Zimba- 
bwean government and the World Bank. Remembering that Woodhouse in 
Chapter 6 points out that ‘rapport can be manipulative’, one might ask 
whether it was manipulation that allowed the Oxfam researchers to bring 
about a change in others’ policy, and if so whether it was justified. Justified 
or not, such a result could not have been achieved without an agency view 
of both the users and the policy-makers. 


THE IMPORTANCE OF THEORY 

Policy-oriented research as the analysis of social processes implies com- 
ing to grips with theory, particularly social theory about how change takes 
place. This means recognizing that contesting theories play their part in 
the policy domain. Thus an important dimension of policy investigation 
resides in understanding the premises and theoretical underpinnings of dif- 
ferent participants in policy debates and practice. 


Despite the fact that some of the differences noted between authors are 
about the role of theory (discovering reality versus constructing realities), 
all of us recognize the importance of locating policy arguments within 
their theoretical traditions so as to get a better grip on the diversity of ac- 
tions undertaken and the ways in which they are justified. This is covered 
explicitly by Barrientos in Chapter 4 on undertaking a literature study, but 
applies equally if one’s way into a new topic or situation is through ‘find- 
ing the woman who knows’. An initial map of a policy area has to include 
not only the main organizations and groups involved and how they relate, 
but also the bases of their actions, and that means their contrasting under- 
lying theoretical views of the world. 


When it comes to pursuing one’s own analysis in order to provide an ex- 
planation and policy recommendations, we have seen the importance of 
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theory emphasized again and again in the chapters of the book. That is to 
say, it is vital to come to a situation armed with clear ideas about what one 
expects to find. Whether they derive directly from recognized, named so- 
cial theories or not, these ideas are the theories which one uses to interpret 
questionnaire survey results (Chapter 6), or to design questions to ‘fire at 
the data’ (Chapter 11) or to shape one’s investigation of a case study (Chapter 
14). At the same time, however, it is also vital to be aware of opposing 
theories and ideas, to allow one’s own ideas deliberately to be challenged 
by them, and to be as open as possible to which gives the best interpreta- 
tion. This is an important aspect of integrity as a researcher. But closing 
one’s mind to opposing views and theories is not a particularly good way 
to be effective either. 


3 > RECONCILING THE DIFFERENCES 

Given such a lot of common ground, surely there is a basis for reconciling 
the differences between authors. We work towards this by first looking at 
the limitations of each of the two main positions identified, and then see 
how pursuing high standards of integrity may imply each taking account 
of the other. 


Our discussion of some of the limitations of the realist and then the social 
constructionist positions is not a full philosophical treatment, but a brief 
consideration in the light of the use of research here to inform policy and 
public action. Then we look at rigour and accountability as two aspects of 
integrity in research practice which cut across any underlying difference 
of philosophical position. 

LIMITS TO REALISM 

No one thinks that it is possible to understand reality completely, or even 
have complete knowledge of the particular small aspect of reality that re- 
lates to the policy issue one is dealing with. In Chapter 3 Hanlon writes: 
‘Not even historians have all the facts’. However, for those who hold to the 
notion of an underlying reality, while we can never have complete knowl- 
edge, so that everything is surrounded by more or less uncertainty, we can 
with effort and good technique approach closer and achieve greater preci- 
sion in our understanding in any given area. The special problem for find- 
ing out fast is to be accurate enough to provide useful results with limited 
time and resources. 


Most critiques of the idea that there is an underlying objective reality cen- 
tre on the impossibility of determining that reality. One argument, men- 
tioned in Chapter 13, concerns cases where causal interlinkages are so 
complex, so long term and so changing that tracing them in detail cannot 
reasonably be done with any degree of certainty. A different argument con- 
cerns those cases where human purpose is involved. These include all cases 
of policy and thus this argument is of particular importance to us here and 
we will pursue it a little further. 
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As noted at several points in this book, investigation itself constitutes in- 
tervention, so that the very act of trying to find out why things were done 
or how policies had certain effects leads those questioned to construct dif- 
ferent meanings for what they have done and to think of different ways in 
which they could react to those policies. Although mathematical and physi- 
cal science analogies are often invoked in this discussion, with reference 
for example to Heisenberg’s uncertainty principle, Gédel’s theorem or chaos 
theory, we would argue that when human purpose is involved uncertainty 
is of a different order altogether, going well beyond physical or probabilistic 
uncertainty to the complete uncertainty of not knowing how a human be- 
ing or human agency will react. (Perhaps, like Mayer, again in Chapter 13, 
it would be better to term this ‘ignorance’ rather than uncertainty.) The 
difficulty in getting a consistent answer from somebody about why they 
acted in a certain way in the past translates into the complete impossibility 
of inferring from past behaviour what they may do in new circumstances, 
because they always have choice and their purposes may change or be 
unclear. And there are many occasions in policy-oriented investigations 
where circumstances are unique. 


Thus, situations involving human choice, and complex interactions between 
choices by different people, are in principle virtually impossible to know 
with any degree of accuracy. There are those who nevertheless hold to the 
view that there is an underlying objective reality, although it is extremely 
difficult to discover it. Perhaps unsurprisingly, such people tend to over- 
emphasize those areas where it is possible to achieve greater apparent cer- 
tainty. One such area is the regular production of secondary data by par- 
ticular institutions through standardized mechanisms. Chapter 11 discusses 
the ‘factuality of data’ and how what is regarded as a ‘fact’ is actually 
socially constructed. Chapter 13 points out that, despite some well-known 
critiques of science that challenge the impartiality and assumptions of the 
people doing the measuring, ‘hard’ quantitative data are still thought of as 
somehow ‘better’ than qualitative data. Chapter 13 goes on to note how 
this attitude had led to the assumption of the superiority of western over 
other forms of knowledge and hence the discounting of traditional knowl- 
edge, in some cases with well-documented detrimental effects. Chapter 9 
puts forward another critique of supposedly objective quantitative data, 
namely the need for involvement and hence subjectivity in order to be- 
come privy to personally sensitive information. 


PROBLEMS WITH SOCIAL CONSTRUCTIONISM 


The alternative notion of research is that it is a question of bringing out and 
recording different accounts from different viewpoints, and then construct- 
ing what appears to be the best story. 


Although many different and even contradictory constructions of reality 
may be valid in this way of thinking, not all accounts are equally so. There 
may be clear evidence against some versions of reality, as when warriors 
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believing a certain oil gives them invincible protection are nevertheless 
mowed down by machine-gun fire. Most evidence, however, is not so in- 
controvertible. In that case, how does one decide what is the best story 
among several possibilities that all seem to fit the (ambiguous) ‘facts’? 


Sometimes, helping those generally silent or disregarded to formulate their 
accounts clearly may appear good in itself. As noted in Chapters 6 and 7, 
using participatory rural appraisal (PRA) or other techniques to allow a 
voice to women, to children and elderly people, and to members of poor 
communities, can be justified directly in the name of empowerment. These 
seldom heard stories may be full of new insights. However, they may also 
be trite, full of inconsistencies, or based on internalized versions of the 
stories of the powerful. Conversely, those in positions of power may also 
be in a good position to see and explain what is going on even though their 
self-justifications are likely to be biased. No technique can automatically 
overcome power relations, and Chapters 6 and 7 also gave examples where 
PRA brings out not the stories of the downtrodden but the ‘official’ expla- 
nation of how things are in a community. Both Woodhouse in Chapter 6 
and Johnson and Mayoux in Chapter 7 also point out that despite its ideal 
portrayal of the sensitive outsider as ‘convenor, catalyst and facilitator’, 
the best insights from PRA come when the researcher uses her or his judge- 
ment to make interpretations and to accept or reject what comes out of the 
various exercises used. They conclude that it is necessary to be explicit 
about a researcher's purpose in undertaking a PRA. 


We are left with the unavoidable need for subjective interpretation by the 
researcher in order to come up with ‘the best story’. Inevitably this means 
the explanation which is most useful for backing up the argument the re- 
searcher or those commissioning the research want to make. There is an 
obvious danger here that an investigation will simply find only what it is 
looking for, and prejudices and biases will be reinforced because contrary 
evidence will be overlooked or ignored. The idea that one can only choose 
between versions of reality on the basis of utility appears to allow those in 
powerful positions to manipulate reality in a way most useful to them. This 
may be acceptable if their power is legitimate, as when in Chapter | re- 
search is called for which will help transport authorities in ‘changing peo- 
ple’s behaviour in a way which is not occurring voluntarily but which would 
be intended to improve their quality of life’. As Hanlon notes in Chapter 3, 
framing different questions will give rise to different stories. 


The integrity of the researcher may appear to be a poor defence against 
these dangers, but it is some defence. It is important to bolster it as much as 
possible, and rigour and accountability are two areas in which this can be 
done. 


RIGOUR 
In the Introduction rigour was defined as “being able to show that one has 
enough evidence to justify one’s conclusions, that the evidence has been 
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obtained properly and that contrary evidence has been sought but either 
not found or found to be relatively unconvincing’. Woodhouse in Chapter 
6 adds the question of how far research findings can be reproduced, cor- 
roborated or applied to other situations. 


Woodhouse also points out that for structured survey methods there is a set 
statistical procedure for rigorously inferring characteristics of a population 
from a sample, but in other cases the key means of achieving rigour are 
triangulation and documentation. Thus although subjective interpretation 
may have been to the fore it will at least be apparent that evidence sought 
from different angles pointed to the same result (triangulation) and be pos- 
sible to look up the record of what evidence was the basis for the interpre- 
tation (documentation). 


The idea of contrastive inference, developed by Mukherjee and Wuyts in 
Chapter 11 was taken up by Thomas in Chapter 14 as a third element in an 
attempt at rigour. Here you deliberately start with more than one set of 
competing ideas, perhaps including those from your theoretical opponents 
as well as your own ideas, and look for evidence of both before deciding 
which is the best interpretation. Combined with triangulation and docu- 
mentation, this offers some defence against accusations of bias or that one’s 
research results are simply what Chapter 14 calls ‘self-fulfilling proph- 
esy’. But subjectivity remains and so no defence is complete. 


ACCOUNTABILITY 

Chataway and Joffe in Chapter 10 consider the question of to whom a 
researcher is accountable. One answer is: to her or his own organization, or 
to whoever commissioned the research. But this ignores the fact that re- 
search is only possible if it obtains information from a great many sources. 
To some extent a researcher is also accountable to those from whom infor- 
mation was obtained. They may have expectations of good resulting from 
their sharing of information. Certainly there could be accusations of false 
pretences if confidences are broken or information is used against those 
who provided it. 


There is also an argument that research results should as far as possible be 
in the public domain, and that since policy research is aimed at informing 
public action then those who produce it should be publicly accountable. 
This position may be hard to uphold when more and more research is com- 
missioned by those who wish to use its results privately. However, for re- 
search aimed at development policy and public action (that is, the kind of 
research where we hope this book helps you to find out fast) a claim of 
public accountability is another defence, this time against pressure to rep- 
resent results in a predetermined way when you have found contrary but 
unwelcome evidence (what Mayer in Chapter 13 calls ‘data that don’t say 
what you want’). 
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4 CHALLENGE: ‘TRYING NOT TO GET IT WRONG’ 


So the idea of searching for the truth is impractical, while telling the best 
story means making subjective judgements which are always subject to 
pressure and cannot be unbiased. Although the two philosophical positions 
may be in basic opposition, in practice one need not choose between them 
but can hang on to elements of each. Thus, since you can never get enough 
evidence to describe all aspects of reality in detail, you may as well choose 
to frame your research question to get the most useful answers; conversely, 
if you have a good idea of the story you want to tell you should try to find 
‘real’ evidence that might disprove it so that if you fail you will be more 
certain that the story stands up. 


The theme which has run throughout the book which is relevant here is that 
of challenge. Trying to find contrary evidence is a way of subjecting your 
ideas to challenge — one which several authors have proposed. Contrastive 
inference — contrasting one set of ideas with another so as to see which 
provides the best explanation of all the evidence — is another form of chal- 
lenge. And if research results are to have any impact there is also the ques- 
tion of how to use them to challenge the people and organizations with 
power to act in a situation. 


Whether or not you hold to a belief in an underlying reality, it is never 
possible to be sure you are right. If you want to be effective as a policy 
investigator, the best you can do is to adopt Hanlon’s slogan of ‘trying not 
to get it wrong’. In Chapter 3 this referred to trying not to reconceptualize 
in a way which those commissioning research would reject. This means 
choosing to tell a story which is acceptable as well as being what the re- 
searcher thinks is the best story. Hanlon’s slogan can also be taken to refer 
to trying not to put forward an explanation which powerful evidence can 
be brought forward to contradict. This means looking for such evidence 
yourself and being prepared to change your story if you find it, rather than 
ignoring it and hoping others will not uncover it. 


So, be challenging and be open to challenge, and — in both senses — try not 
to get it wrong! 
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Finding Out Fast will provide readers with key skills 
and approaches for research designed to inform policies, 
particularly on development. 


Recognizing that policy decisions are typically made 
under pressure of time and on the basis of incomplete 
data or with limited resources with which to obtain 
information, the authors provide guidance on how to 
locate, evaluate and use relevant information, fast. 


They explain and critically review a variety of research 
methods for ‘finding out fast’, in the belief that these 
methods can still be rigorous. Finding Out Fast will assist 
readers to become competent investigators, to 
understand how to use research more effectively and 
how to commission and critically evaluate research done 
by others. / 


This is essential reading for development managers in 
non-governmental organizations (NGOs) and public 
sector agencies and students of development 
management and development studies more generally. 
However, the ideas and approaches will also appeal to 
anyone involved in policy making and research with 
developmental goals. if 
It is a course text for two courses in the Open University’s 
Global Programme in Development Management: 
Capacities for Managing Development (TU870) and the 
Development Management Project (TU874). 
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